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ABSTRACT 


We  present  modifications  of  standard  sequential  decoding  algorithms  in 
an  attempt  to  operate  at  rates  greater  than  R0,  the  computational  cutoff  rate. 
We  call  the  new  algorithms  sequential  decoding  with  reordering  (SDR)  algo¬ 
rithms.  They  observe  the  received  message  at  the  channel  output  and  use  this 
information  to  reorder  the  digits  in  the  codeword  tree.  The  resulting  tree  is 
then  searched  by  a  sequential  decoder;  the  goal  of  reordering  is  to  obtain  a 
tree  that  is  easy  to  search.  However,  codeword  trees  associated  with  convolu¬ 
tional  codes  cannot  be  reordered  and  still  retain  their  uniform  structure  and 
slow  growth.  For  this  reason,  we  use  low-density  codes,  a  class  of  block 
codes. 

The  SDR  algorithms  are  presented  for  the  binary  erasure  and  binary  sym¬ 
metric  channels.  Simulation  results  suggest  that  at  high  code  rates,  the  algo¬ 
rithms  can  be  used  at  rates  nearly  equal  to  or  greater  than  RQ. 

Because  of  differences  between  codeword  trees  for  low-density  codes  and 
convolutional  codes,  we  present  a  new  sequential  decoding  algorithm  designed 
for  low-density  codes.  We  also  present  an  SDR  algorithm  that  can  be  used  as 
a  soft  decision  decoder  on  channels  with  side  information  and  channels  with 


real-valued  outputs. 
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INTRODUCTION 


1.1  Introduction 

We  present  modifications  of  standard  sequential  decoding  algorithms  in 
an  attempt  to  operate  at  rates  greater  than  the  computational  cutoff  rate. 
Sequential  decoding  is  a  general  method  for  decoding  tree  codes,  including 
the  important  class  of  convolutional  codes.  It  was  invented  by  Wozencraft  in 
1957  and  subsequently  modified  by  Fano  [4]  and  others.  For  a  good  introduc¬ 
tion  to  this  topic,  see  Section  7.2  of  Clark  and  Cain  [3]. 

The  amount  of  computation  performed  by  a  sequential  decoder  depends 

on  the  channel  noise.  In  particular,  a  long  burst  of  noise  will  require  a  great 

deal  of  computation.  This  is  illustrated  in  the  following  example,  taken  from 

[8].  Suppose  we  have  a  binary  erasure  channel  with  transition  probabilities  as 

shown  in  Figure  1.1.  Suppose  further  that  we  use  a  convolutional  code  with 

rate  R.  If  the  first  L  received  symbols  are  erasures,  each  path  with  length  L 

symbols  in  the  codeword  tree  will  appear  equally  likely  to  the  decoder  until 

LR 

the  paths  are  extended  further  into  the  tree.  There  are  approximately  2 
such  paths,  and  each  path  has  probability  1/2  of  being  extended  before  the 
decoder  finds  the  correct  one.  Thus,  given  L  initial  erasures,  the  expected 
number  of  paths  searched  by  the  decoder  is  2  /2. 

A  long  burst  of  noise  will  cause  a  sequential  decoder  to  perform  a  great 
deal  of  computation  even  on  more  general  communication  channels.  For 
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example,  consideration  of  bursts  allowed  Jacobs  and  Berlekamp  [8]  to  prove 
the  following  result.  Consider  a  discrete  memoryless  channel  with  input 
alphabet  {0,1, ...,/C — 1},  output  alphabet  {0,1,...,/— 1},  and  transition  proba¬ 
bilities  P(j  \k).  If  one  uses  a  code  with  rate  R  satisfying  R>Rp,  then 

lim  E(cnP)  =  oo,  (i.i) 

rt— ►  CO 

where 

E0{P) 

R>  = - 

P 

Eq (p)  =  the  concave  hull  of  EQ(p) 

E0(p)  =  Gallager’s  reliability  exponent 


f  7-1 

K- 1 

!+/> 

max  ■ 

-logE 

se(*)p(/ \k),/(Ur) 

Q 

^  0 

■  k-0 

> 

cn  =  the  amount  of  computation  per  symbol 
required  to  decode  the  first  n  symbols 

This  result  is  valid  for  all  p>0,  for  all  tree  codes,  which  include  the  class  of 
convolutional  codes,  and  for  all  sequential  decoders.  It  is  derived  by  consid¬ 
ering  the  effect  of  noise  bursts  with  specified  length  and  severity.  These  noise 
bursts  are  shown  to  occur  sufficiently  often  and  cause  the  decoder  to  search 
enough  of  the  codeword  tree  to  result  in  the  unbounded  moments  of  compu¬ 
tation  described  above.  Note  that  Arikan  [1]  has  obtained  an  improved 
bound  for  the  case  p  =  1.  His  result  shows  that  E#’/>  )  =  oo  for  all  R>E0(  1), 
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which  differs  from  the  result  given  above  by  using  E0(p)  instead  of  its  concave 
hull. 

For  future  reference,  note  that  £0(1),  which  is  a  function  of  the  channel, 
is  traditionally  denoted  by  RQ.  (This  presentation  follows  [1].)  The  computa¬ 
tional  cutoff  rate  for  sequential  decoding,  Rcomp,  is  defined  to  be  the 
supremum  of  code  rates  for  which  lim  E(cn)<  oo.  From  Arikan’s  result  and 

rt— ♦  OO 

previous  work  (p.  279  of  Gallager  [5]),  it  is  known  that  Rcomp  —  R0-  We  use 
R0  as  a  reference  point  to  gauge  the  performance  of  the  decoding  algorithms 
presented  in  this  work. 

The  preceding  discussion  leads  one  to  consider  what  would  happen  if 
bursts  of  noise  could  somehow  be  eliminated.  Would  it  be  possible  to  have 
bounded  moments  of  computation  even  at  rates  above  Rp ?  One  way  to  limit 
the  occurrence  and  duration  of  bursts  is  to  adaptively  reorder  the  codeword 
tree,  that  is,  to  change  the  order  of  the  digits  used  to  generate  the  tree.  How¬ 
ever,  one  cannot  significantly  reorder  the  codeword  tree  associated  with  a 
convolutional  code  and  still  obtain  a  tree  that  is  practical  to  search. 

Instead  of  using  convolutional  codes,  one  can  use  low-density  codes. 
Low-density  codes,  devised  by  Gallager  [6],  [7],  are  linear  block  codes  charac¬ 
terized  by  three  parameters.  A  binary  ( n,j,k )  low-density  code  has  block- 
length  n,  and  a  parity  matrix  with  exactly  j  l’s  in  each  column  and  k  l’s  in 
each  row.  All  low-density  codes  considered  in  this  paper  are  binary. 
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Typically,  j  and  k  are  much  smaller  than  n,  resulting  in  a  sparse  parity 
matrix.  In  addition,  choosing  small  values  for  j  and  k  causes  low-density 
codes  to  have  the  following  two  features.  First,  given  any  ordering  of  the  par¬ 
ity  checks  used  to  define  a  low-density  code,  one  can  generate  a  codeword 
tree  that  grows  slowly  compared  to  the  growth  for  a  dense  parity  matrix. 
Second,  different  parity  check  orderings  yield  different  codeword  trees. 

These  two  features  are  used  by  the  decoding  algorithms  presented  in  this 
thesis.  The  algorithms,  which  we  call  sequential  decoding  with  reordering 
(SDR)  algorithms,  all  have  the  structure  shown  in  Figure  1.2.  The  ordering 
algorithm  observes  the  channel  output  and  uses  this  information  to  choose  a 
parity  check  ordering.  The  sequential  decoder  then  searches  the  correspond¬ 
ing  codeword  tree  to  obtain  the  decoder  output. 

A  method  for  generating  codeword  trees  for  low-density  codes  with  given 
parity  check  orderings  is  presented  in  Section  1.2.  Some  differences  between 
these  trees  and  codeword  trees  for  convolutional  codes  are  discussed.  Sec¬ 
tion  1.3  summarizes  some  of  the  properties  of  low-density  codes.  This 
includes  a  result  concerning  their  asymptotic  minimum  distance,  obtained  by 
Gallager  [7].  Section  1.4  describes  some  known  decoding  algorithms  for  low- 
density  codes. 

Chapter  2  discusses  an  SDR  algorithm  for  binary  erasure  channels.  It 
includes  the  results  of  using  this  algorithm  on  a  simulated  communication 
channel. 


Figure  1.2.  The  general  structure  of  SDR  algorithms 
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Chapter  3  contains  SDR  algorithms  for  binary  symmetric  channels. 
Several  ordering  algorithms  are  presented.  Because  of  the  differences 
between  codeword  trees  for  low-density  codes  and  convolutional  codes,  a  new 
sequential  decoding  algorithm  is  presented  as  well.  Simulation  results  are 
presented  also. 

Chapter  4  contains  conclusions  and  possible  directions  for  further 
research.  A  decoding  algorithm  for  memoryless  channels  with  binary  input 
and  arbitrary  output  is  presented.  In  particular,  it  is  applicable  to  additive 
white  Gaussian  noise  channels  and  Rayleigh  fading  channels.  This  algorithm 
allows  one  to  use  soft  decision  decoding. 

Throughout  this  work  we  use  the  following  terminology.  The  vector 
x  =  (*Xi  is  the  transmitted  codeword,  y  =  is  the  channel  output,  or 

received  message,  and  x  =  is  the  decoder  output,  where  n  is  the  block- 

length  of  the  code  being  used. 

1.2  Codeword  Trees  for  Low-Density  Codes 

In  this  section,  we  present  a  method  for  generating  codeword  trees  for 
low-density  codes  and  discuss  some  properties  of  these  trees. 

The  following  description  applies  to  a  low-density  code  with  parameters 
n,j,  and  k.  To  generate  a  codeword  tree,  one  must  first  assign  an  ordering 
to  the  parity  checks  used  to  define  the  code.  Any  ordering  will  yield  a  code¬ 
word  tree,  and  at  this  point  we  assume  the  ordering  is  arbitrary.  Algorithms 
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for  choosing  an  ordering  are  given  in  Chapters  2  and  3.  Each  parity  check 
corresponds  to  a  level  in  the  codeword  tree.  Since  there  are  n  j/k  parity 
checks,  there  will  be  nj/k  levels  in  the  tree.  As  with  any  tree  code,  the 
nodes  of  the  tree  correspond  to  partially  filled  codewords.  The  root  node  is 
completely  unfilled  and  the  nodes  at  the  final  level  correspond  to  complete 
codewords. 

The  first  level  of  the  tree  is  generated  by  assigning  values  to  the  digits 
involved  in  the  first  parity  check.  A  different  node  is  created  for  each  assign¬ 
ment  that  satisfies  the  parity  check.  (See  Figure  1.3.)  Since  there  are  k  digits 

k  —1 

involved  in  each  parity  check,  there  will  be  2  nodes  in  the  first  level.  Each 
of  these  nodes  is  extended  to  the  second  level  by  assigning  values  to  the  digits 
involved  in  the  second  parity  check.  Again,  this  is  done  in  all  ways  that 
satisfy  the  new  parity  check.  However,  if  some  digits  are  involved  in  both  the 
first  and  second  parity  checks,  the  values  that  they  were  assigned  in  the  first 
level  are  used  without  change  in  the  second  level.  In  general,  a  digit  is 
assigned  a  value  only  at  one  level  in  the  tree  and  has  the  same  value  in  all  sub¬ 
sequent  levels. 

This  procedure  is  continued  until  all  the  parity  checks  have  been  used  to 
generate  levels  in  the  tree.  A  special  case  arises  if  all  the  digits  involved  in  a 
parity  check  have  already  been  assigned  values  in  previous  levels.  Then  the 
new  level  is  generated  by  extending  only  the  nodes  that  satisfy  the  new  parity 
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check.  When  the  procedure  is  completed,  the  final  level  contains  a  list  of  all 
the  codewords  in  the  code. 

This  procedure  can  be  used  with  any  linear  block  code.  In  the  general 

case,  however,  there  is  no  limit  on  the  number  of  digits  involved  in  a  parity 

check,  other  than  the  blocklength  n.  A  typical  parity  check  may  involve  nf 2 

digits.  If  this  is  the  first  parity  check  used  to  generate  the  codeword  tree, 

there  will  be  2^2-1  nodes  in  the  first  level.  Even  for  moderate  n,  this  is  too 

large  to  be  searched  by  a  practical  decoder.  However,  in  an  ( n,j,k )  low- 

density  code,  each  parity  check  used  to  define  the  code  involves  exactly  k 

digits.  The  parameter  k  is  typically  small,  and  may  be  chosen  independently 

of  n .  All  of  the  low-density  codes  used  in  this  study  have  k  less  than  or  equal 

to  8.  (Note  that  this  k  is  not  the  same  as  the  number  of  information  symbols 

associated  with  each  codeword,  which  is  also  commonly  denoted  by  k.)  As 

will  be  shown  below,  codeword  trees  for  low-density  codes  have  fanout  lim- 
k—\ 

ited  by  2  ,  and  typically  the  fanout  is  much  less. 

One  property  of  the  codeword  trees  obtained  using  the  method  given 
above  is  the  variability  of  their  fanout,  or  growth  rate.  This  contrasts  with 
codeword  trees  for  convolutional  codes,  which  grow  at  the  same  rate  at  every 
level.  To  characterize  the  growth  rate,  we  first  present  some  definitions.  The 
set  of  digits  involved  in  a  parity  check  is  referred  to  as  a  parity  check  set. 
Given  an  ordering  of  the  parity  checks,  each  parity  check  set  is  partitioned 
into  two  groups,  the  n-set  and  the  o-set.  The  n-set,  which  stands  for  new  set, 
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is  made  up  of  the  digits  not  contained  in  any  previous  parity  check  set,  and 
the  o-set,  or  old  set,  contains  the  remaining  digits.  Recall  that  each  level  in 
the  codeword  tree  is  associated  with  a  unique  parity  check.  The  set  of  digits 
that  are  assigned  values  at  a  given  level  in  the  tree  are  the  digits  in  the 
corresponding  n-set. 

Given  a  codeword  tree  for  an  ( n,j,k )  low-density  code,  let  ni  be  the 
number  of  digits  in  the  n-set  associated  with  level  i  in  the  tree.  Also,  let  Nt 
be  the  total  number  of  nodes  at  level  i,  with  N0  =  1.  Then,  for  1<  i  <  n  j/k, 

(i)  If  the  i‘h  parity  check  is  independent  of  the  preceding  parity  checks, 

N I  -  2 (1.2) 

(ii)  Otherwise, 

(1.3) 

In  particular,  t  m 

^  <  2‘"lWi_1.  (1.4) 

To  see  why  this  is  true,  suppose  n(>  1.  Then  there  are  2”'  ways  to  assign 
values  to  the  ni  digits  filled  in  at  level  i ,  and  half  these  assignments  will  satisfy 
the  corresponding  parity  check.  Since  in  this  case  parity  check  i  must  be 
independent  of  the  preceding  ones,  case  (i)  applies  and  (1.2)  is  valid.  On  the 
other  hand,  suppose  ni  =  0  and  parity  check  i  is  independent  of  the  preceding 
ones.  Then,  in  going  from  level  i—  1  to  level  i,  the  dimension  of  the  set  of 
codeword  fragments  represented  in  the  tree  will  decrease  by  one.  Thus 
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Nt  =  ^V,_1/2  and  (1.2)  is  again  valid.  If  parity  check  i  is  not  independent  of 
the  preceding  ones,  then  ni  must  be  zero,  thus  Ni  cannot  be  greater  than 
iV(_r  Furthermore,  every  codeword  fragment  satisfying  the  first  i—  1  parity 
checks  will  necessarily  satisfy  parity  check  /,  thus  ZV-=  This  establishes 

(1.3).  Since  every  parity  check  set  contains  k  digits,  nt<k  for  all  i.  This 
yields  (1.4)  and  concludes  the  proof. 

This  result  implies  that  codeword  trees  for  low-density  codes  tend  to 

grow  quickly  near  the  beginning  of  the  tree  and  more  slowly  toward  the  end. 

This  occurs  because  the  growth  rate  at  a  given  level  depends  on  the  size  of 

the  corresponding  n-set.  At  a  level  close  to  the  origin,  few  digits  will  have 

had  a  chance  to  appear  in  previous  parity  check  sets,  hence  the  corresponding 

n-set  tends  to  be  large.  The  opposite  situation  holds  near  the  end  of  the  tree. 

As  a  result,  ni  will  roughly  be  a  decreasing  function  of  i.  This  is  not  a  hard 
* 

and  fast  rule,  but  a  good  qualitative  description  that  holds  regardless  of  which 
parity  check  ordering  is  used. 

In  addition  to  the  variable  growth  rate,  codeword  trees  for  low-density 
codes  and  those  for  convolutional  codes  differ  in  another  important  way. 
With  a  low-density  code,  two  different  subtrees  descending  from  a  common 
parent  node  can  be  identical  for  many  levels.  This  happens  because  the  set  of 
digit  values  assigned  to  a  node’s  children  depends  only  on  the  parity  of  the 
child  level’s  o-set.  As  a  result,  two  subtrees  with  a  common  parent  node  will 
agree  on  the  values  they  assign  to  codeword  digits  until  a  level  is  reached 
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where  the  o-set  in  one  tree  has  different  parity  than  the  o-set  in  the  other 
tree.  To  determine  where  such  a  level  can  occur,  consider  the  root  nodes  of 
the  two  subtrees.  The  labels  of  these  two  nodes  will  differ  only  in  the  values 
assigned  to  the  digits  in  the  n-set  of  the  level  containing  the  root  nodes. 
Denote  this  n-set  by  S.  Then  the  first  level  at  which  the  two  trees  can  differ 
must  have  an  o-set  that  contains  one  or  more  digits  in  S. 

This  property  can  significantly  affect  the  behavior  of  a  sequential 
decoder.  If  the  decoder  diverges  from  the  correct  path  in  the  tree,  it  may  not 
be  able  to  detect  its  error  for  many  levels.  In  this  case,  backtracking  one 
level  at  a  time  would  be  very  inefficient.  For  example,  when  using  a  (396,5,6) 
low-density  code  and  a  typical  parity  check  ordering,  ten  percent  of  the  digits 
will  have  50  or  more  levels  between  their  appearance  in  an  n-set  and  their  first 
appearance  in  an  o-set.  For  this  reason,  a  new  sequential  decoding  algorithm 
is  presented  in  Section  3.3. 

1.3  Properties  of  Low-Density  Codes 

This  section  contains  a  summary  of  some  properties  of  low-density 
codes.  They  are  included  here  for  general  interest  and  because  most  will  be 
referred  to  elsewhere  in  this  work.  The  following  results  were  obtained  by 
Gallager  [6],  [7]  unless  stated  otherwise. 

Recall  that  a  binary  (n,  j ,k)  low-density  code  is  a  linear  block  code  with 
blocklength  n  and  a  parity  matrix  with  exactly  j  l’s  in  each  column  and  k  l’s 


in  each  row.  Note  that  this  k  differs  from  the  quantity  usually  denoted  by  k 
in  the  context  of  coding  -  namely,  the  number  of  information  symbols  associ¬ 
ated  with  each  codeword.  This  definition  does  not  define  the  codes  uniquely; 
there  may  be  many  low-density  codes  for  a  given  value  of  (n,  j ,k).  Figure  1.4 
contains  the  parity  matrix  for  a  (12,3,4)  low-density  code. 

Low-density  codes  do  not  exist  for  all  values  of  n,  j ,  and  k.  Specifically, 
the  product  nj  must  be  a  multiple  of  k.  To  see  why,  let  L  be  the  number  of 
rows  in  a  low-density  code’s  parity  matrix.  Counting  by  rows,  there  are  a 
total  of  Lk  l’s  in  the  matrix;  counting  by  columns  yields  a  total  of  nj.  Since 
these  expressions  must  be  equal,  we  have 

Lk  =  nj  ,  (1.5) 

and  thus  nj  is  a  multiple  of  k. 

Furthermore,  all  the  codes  considered  in  this  work  have  n  a  multiple  of 
k.  This  restriction  is  followed  by  Gallager  as  well.  The  blocklength  is  con¬ 
strained  in  this  way  because  it  is  then  quite  easy  to  construct  a  low-density 
code  for  a  given  set  of  parameter  values.  The  method  of  construction  is 
described  later  in  this  section.  This  is  not  a  severe  restriction  because  k  is 
typically  small  -  k  is  less  than  10  for  all  the  codes  used  in  this  work. 

Equation  (1.5)  implies  that  L,  the  number  of  rows  in  the  parity  matrix, 
equals  nj/k.  For  this  reason,  j  is  always  chosen  to  be  less  than  k.  Other¬ 
wise  L  would  be  greater  than  n,  which  is  undesirable,  because  if  n  of  the 
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n=1  2 

.  j=3. 
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1000 
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0010 

Figure  1.4.  The  parity  matrix  of  a  (12,3,4)  low-density  code. 
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parity  checks  were  linearly  independent,  the  code  rate  would  be  zero. 

Since  the  rank  of  the  parity  matrix  equals  n(l  —  R),  where  R  is  the  code 
rate  in  bits,  and  since  the  rank  of  any  matrix  cannot  be  greater  than  the 
number  of  rows,  we  have 


n(l-R)<  —  .  (1.6) 

k 

Equivalently, 

R  >  1  -  -  .  (1.7) 

k 


This  relationship  is  satisfied  with  equality  if  and  only  if  all  the  rows  of  the  par¬ 
ity  matrix  are  linearly  independent,  and  this  does  not  hold  for  the  low-density 
codes  used  in  this  study.  However,  the  parity  checks  used  to  define  these 
codes  are  chosen  randomly,  and  thus  one  would  expect  the  number  of  linearly 
independent  parity  checks  to  be  close  to  the  total  number.  For  this  re.ic  >n, 
the  quantity  1  —  j/k  is  referred  to  as  the  designed  code  rate.  In  any  case, 
(1.7)  remains  a  valid  lower  bound. 

Gallager  studied  the  minimum  distance  properties  of  a  random  ensemble 
of  these  codes.  He  obtained  an  upper  bound  to  the  minimum  distance  distri¬ 
bution  function  P(dmin  <  bn).  Note  that  Gallager’s  ensemble  does  not  con¬ 
tain  all  possible  low-density  codes,  but  only  those  obtainable  by  the  construc¬ 
tion  process  described  below.  For  fixed  n,  j,  and  k,  where  n  is  a  multiple  of 
k,  the  ensemble  is  defined  in  the  following  way.  Start  with  A,  the  ( n/k )  X  n 
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matrix  shown  in  Figure  1.5.  Row  i  of  A  contains  l’s  in  positions  (i—  l)k 
through  ik  and  0’s  elsewhere.  To  choose  a  code  from  the  ensemble,  create  j 
new  matrices  by  permuting  the  columns  of  A  j  times.  These  matrices  are 
created  independently,  with  each  possible  permutation  having  equal  probabil¬ 
ity.  They  are  placed  together  to  form  a  single  ( nj/k )  Xn  matrix,  the  parity 
matrix  of  the  chosen  code.  Figure  1.4  contains  an  example  of  the  parity 
matrix  of  a  code  chosen  from  the  ensemble  of  (12,3,4)  low-density  codes. 
Each  submatrix  contains  exactly  k  l’s  in  each  row  and  a  single  1  in  each 
column,  so  the  procedure  yields  a  valid  ( n,j,k )  low-density  code.  Gallager’s 
ensemble  differs  slightly  from  the  one  just  described  in  that  the  first  subma¬ 
trix  is  always  chosen  to  be  an  unpermuted  copy  of  A .  However,  both  ensem¬ 
bles  have  the  same  minimum  distance  distribution  function. 

Gallager’s  upper  bound  to  P(dmin  <  <5/i)  is  fairly  complicated  and  will  not 
be  presented  (see  Theorem  2.4  of  [7]).  Of  interest  here  is  its  behavior  as  n 
increases,  with  j  and  k  fixed.  When  j  >  3,  the  bound  converges  to  a  unit 
step  at  8jk,  where  8-k  is  the  only  strictly  positive  zero  of  the  function  B{\). 
This  function  is  given  by 


B(\)  =  (y-l)/t  (X)- 


k  L 


M(s)- (-  (&— l)ln2 \+js\  , 


(1.8) 


where 


h(\)  =  —  XlnX  —  (1— X)ln(l— X), 


(1.9) 


8 


n=1  2,  k  =  4 

1111  0000  0000 
0000  1111  0000 
0000  0000  1111 


Figure  1.5.  The  matrix  A  ,  for  n  =  12  and  k  =  4. 


19 

M*)-  In  2~*  [(1  +  e*)A+  (1—  e1)*] ,  (110) 

and  s  satisfies 

=  \k  .  (1.11) 


This  means  that  for  large  n,  almost  all  codes  in  the  ensemble  have  minimum 
distance  dmiQ  close  to  or  greater  than  6jkn.  This  linear  increase  with  n  of  the 
typical  minimum  distance  is  a  good  property  to  have  and  it  is  uncommon 

among  other  classes  of  codes.  In  fact,  the  Justesen  codes  are  the  only  known 

d  . 

min 

class  of  codes  with  an  explicitly  defined  construction  for  which  the  ratio  - 

n 

is  bounded  away  from  zero,  when  the  rate  R  is  fixed  and  n  goes  to  infinity 
(see  Section  7.9  of  Blahut  [2]). 

For  the  case  j  =  2,  Gallager  showed  that 


n 

21n 

2 


<*min  <  2  + 


Thus,  when  j  — 2  and  k  is  fixed. 


In(fc-l) 

— ►  0  as  n  goes  to  infinity. 


(1.12) 


For  this 


n 

reason,  low-density  codes  with  j  =  2  are  not  considered  in  this  work.  When 
j  —  1,  dmia  =  2,  thus  codes  with  this  value  of  j  are  not  considered  either. 


Another  significant  property  of  Gallager’s  upper  bound  to  P(dmin 
is  that  for  large  n  it  tends  to  have  a  small  step  at  8  =  2 jn.  The  amplitude  of 
this  step  is  0(n  ^  2^).  This  fact  led  Gallager  to  construct  an  expurgated 
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ensemble  of  low-density  codes  by  throwing  out  the  half  with  the  smallest 
minimum  distance.  Otherwise,  the  codes  with  dmia=2  dominated  his  bound 
on  the  ensemble  average  of  the  decoding  error  probability  when  a  maximum 
likelihood  decoder  was  used.  In  order  to  avoid  codes  with  dmin=2,  all  the 
codes  used  in  this  study  were  constrained  to  have  the  following  property: 

No  two  parity  check  sets  contain  more  than  one  digit  in  common. 

The  codes  used  by  Gallager  in  his  simulations  were  chosen  to  satisfy  this  pro¬ 
perty  as  well.  The  term  “parity  check  set”  denotes  the  set  of  digits  involved 
in  a  parity  check.  Note  that  the  parity  check  sets  referred  to  in  the  property 
are  those  contained  in  the  parity  matrix  used  to  define  the  code  -  the  pro¬ 
perty  does  not  apply  to  derived  parity  checks  formed  by  taking  linear  combi¬ 
nations  of  rows  of  the  parity  matrix. 

This  property  also  ensures  the  validity  of  the  first  step  of  Gallager’s 
decoding  algorithm  for  low-density  codes,  described  in  Section  1.4  of  this 
thesis.  The  results  of  the  first  step  are  used  to  generate  the  reliability  infor¬ 
mation  utilized  by  the  decoding  algorithms  presented  in  Chapter  3.  An  algo¬ 
rithm  that  generates  the  low-density  codes  used  in  this  thesis  is  described  in 


Section  A.l. 
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1.4  Other  Decoding  Algorithms  for  Low-Density  Codes 

This  section  contains  a  summary  of  some  known  decoding  algorithms  for 
low-density  codes.  The  first  algorithm  is  due  to  Gallager  [6],  [7].  The  first 
step  of  this  algorithm  is  incorporated  into  some  of  the  decoding  algorithms 
presented  in  Chapter  3.  Two  algorithms  due  to  Zyablov  and  Pinsker  [9],  [10] 
are  included  also.  One  is  for  the  binary  erasure  channel,  and  the  other  is  for 
the  binary  symmetric  channel. 

Gallager  presented  two  related  decoding  algorithms  for  low-density  codes 
([6],  or  pp.  41-45  of  [7]).  One  algorithm  can  be  used  only  on  a  binary  sym¬ 
metric  channel,  that  is,  a  channel  with  both  binary  input  and  binary  output. 
The  second  algorithm  can  be  used  on  any  binary-input  memoryless  channel; 
in  particular,  it  can  be  used  on  a  channel  with  real-valued  outputs,  such  as  the 
additive  white  Gaussian  noise  channel.  The  first  algorithm  is  simpler  but  less 
powerful  than  the  second  algorithm,  and  only  the  second  algorithm  will  be 
described  here. 

Recall  the  following  terminology.  The  vector  x  =  (*,•)"- i  is  the  transmit¬ 
ted  codeword  and  y  =  (y/)”-1  is  the  channel  output,  where  n  is  the  block- 
length.  In  the  following  discussion,  the  channel  is  assumed  to  be  memoryless 
and  able  to  accept  binary  inputs,  but  is  otherwise  unconstrained.  The  code 
being  used  is  a  low-density  code  with  parameters  ( n,j,k ).  The  notation  that 
follows  differs  from  Gallager ’s. 
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Gallager’s  algorithm  is  iterative.  Because  it  is  difficult  to  describe,  a  sim¬ 
plified  version  is  presented  first.  The  first  iteration  starts  with  the  n  dimen¬ 
sional  vector  whose  components  are  defined  by 

z/0J  =  P(*,  =  i  |y(),  (1.13) 


Note  that  yi  is  the  raw  channel  output,  and  is  not  necessarily  binary  valued. 
Subsequent  values  of  z  are  computed  using  the  following  formula. 


1 — z,<p+1) 

-(P+1) 


1—z 


( p ) 


.( p ) 


j 

n 

2-1 


k- 1 

i+na-2zx°’)) 

m- 1 
k- 1 

i-na-^x0”) 

m— 1 


1  </  <n,  p>  1,  (1.14) 


where  z ^  is  the  component  of  z^  corresponding  to  the  m-th  digit  other  than 
xi  in  the  /-th  parity  check  containing  xr  The  algorithm  continues  until  each 
component  of  z ^  tends  toward  either  0  or  1.  The  decoder  output  is  then 
(x- )”_ j ,  where  is  the  value  toward  which  is  converging. 


To  interpret  this  procedure,  consider  the  parity  check  tree  shown  in  Fig¬ 
ure  1.6.  The  root  node  represents  xi  and  is  said  to  be  in  the  first  level.  The 
nodes  in  the  second  level  of  the  tree  represent  all  the  digits  that  are  “linked” 
with  x(,  where  two  digits  are  linked  if  they  occur  in  a  common  parity  check. 
Thus,  the  j  edges  leaving  the  root  node  represent  the  j  parity  checks  contain¬ 
ing  x-t,  and  each  edge  leads  to  k—  1  nodes.  Similarly,  nodes  in  the  third  level 
represent  digits  that  are  linked  with  digits  in  the  second  level.  However,  only 


three  levels  of  the  parity  check  tree  rooted  at  x-,  for 
erms  in  parentheses  refer  to  ( l,m )  values  used  in  the 
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1  edges  emanate  from  a  node  at  level  two.  This  happens  because  for  a 
digit  at  level  two,  the  parity  check  containing  both  it  and  the  root  node 
already  has  its  digits  listed  in  the  tree.  The  parity  check  tree  is  extended  in 
this  fashion,  and  continues  indefinitely.  From  the  second  level  on,  each  node 
has  1  sets  of  k—  1  children.  However,  note  that  any  given  digit  will  appear 
more  than  once  in  the  tree,  and  when  the  root  node  appears  again,  the  tree 
essentially  repeats  itself. 

Then  (1.14)  may  be  interpreted  as  follows.  By  definition,  is  the  pro¬ 
bability  that  xi  equals  1,  given  the  channel  output  yr  In  addition,  Gallager 
showed 

|  {y,,l6St}),  (1.15) 

where  54-  is  the  set  of  indices  of  the  digits  in  the  first  two  levels  of  the  parity 
check  tree  rooted  at  xr  Thus,  from  the  viewpoint  of  digit  i,  z equals  the 
probability  that  xi  equals  one,  conditioned  on  the  channel  output  at  the  first 
level  of  the  parity  check  tree,  and  z^  equals  the  probability  of  the  same 
event,  conditioned  on  the  channel  output  at  the  first  two  levels  of  the  parity 
check  tree.  This  may  be  generalized;  the  algorithm  is  designed  so  that  each 
iteration  expands  the  conditioning  set  to  include  one  more  level  of  the  parity 
check  tree. 

However,  the  algorithm  as  described  above  will  not  achieve  this  goal. 
Specifically,  the  interpretation  breaks  down  for  zK  ’  and  subsequent  iterations. 
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This  occurs  because  a  necessary  independence  assumption  about  the  digits  in 
the  conditioning  set  breaks  down.  As  a  concrete  example,  suppose  that  for  a 
given  low-density  code,  the  parity  check  tree  rooted  at  digit  15  has  digit  20  in 
the  second  level.  Then,  not  only  will  the  value  of  z|^  depend  on  ,  but 
Z^o  will  depend  on  z^.  Therefore,  the  value  of  zj^,  which  depends  on  z$q  , 
will  in  turn  depend  on  z  ^ . 

To  avoid  this  problem,  Gallager’s  decoding  algorithm  differs  from  the 
above  description  in  the  following  way.  Each  digit  i  is  associated  with  j  pro¬ 
bability  values,  denoted  by  Z^\  1)  through  Z^\j),  instead  of  the  single  value 
Z^\  Initially,  these  quantities  are  the  same; 

Zi°\>)  “  =  P(*j“  1  bj).  !<></•  (1.16) 

However,  the  updates  differ.  Each  z^\t)  ignores  one  of  the  parity  checks 
involving  digit  / .  In  other  words, 


l-ri°,+1)(t) 

^+1,(0 


i-^V) 

i<l<i 


k- 1 

i+n<i-2^0’)(O) 

m-1 

i-n(i-2??”('x)) 

m-l 


1  <  i  <  n ,  p>  1 . 

(1.17) 


This  differs  from  (1.14)  in  that  the  f-th  term  is  omitted  in  the  outermost  pro¬ 
duct.  Also,  fx  is  chosen  so  that  the  update  for  z^\t>)  is  the  one  that  ignores 
the  parity  check  containing  digit  i.  The  algorithm  stops  when,  for  all  digits 
!</</»,  all  j  of  the  quantities  Z^\l)  through  Z^\j)  converge  to  zero  or 
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one.  If  these  quantities  do  not  converge,  a  decoding  failure  is  declared. 

Even  with  this  modification,  the  independence  assumption  will  be 
violated  at  a  level  in  the  parity  check  tree  where  a  digit  appears  for  a  second 
time.  As  mentioned  previously,  this  must  happen  eventually.  However,  Gal- 
lager  reasons  that  the  dependencies  have  a  minor  effect  and  tend  to  cancel 
each  other  out.  Furthermore,  after  several  iterations,  the  value  of  z ^  may  be 
interpreted  as  the  initial  value  z^  corresponding  to  a  received  sequence  that 
is  easier  to  decode  than  the  original  one.  Therefore,  the  iterations  are  contin¬ 
ued  even  after  dependencies  occur. 

For  this  algorithm,  Gallager  showed  that  the  average  number  of  iterations 
required  to  decode  is  O(loglogrt).  He  also  obtained  loose  bounds  on  the 
probability  of  decoding  error  when  using  a  binary  symmetric  channel.  When 
j  =3,  the  probability  of  decoding  error  is  bounded  by  some  small  negative 
power  of  n,  and  when  j  >3,  it  is  bounded  by  an  exponential  of  a  root  of  n. 
These  bounds  are  known  to  be  valid  only  when  the  crossover  probability  is 
sufficiently  small;  how  small  it  must  be  depends  on  j  and  k.  Gallager 
hypothesized  that  the  probability  of  decoding  error  actually  decreases 
exponentially  in  n . 

Apart  from  its  general  interest  as  a  decoding  algorithm  for  low-density 
codes,  Gallager’s  algorithm  is  described  here  because  of  its  use  in  some  of  the 
decoding  algorithms  presented  in  this  thesis.  The  first  iteration  of  Gallager’s 
algorithm  is  used  to  generate  a  digit  reliability  measure  for  some  of  the  binary 
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symmetric  channel  decoding  algorithms  presented  in  Chapter  3.  Only  a  single 
value  is  generated  for  each  digit,  hence  the  simplified  update  Equation  (1.14) 
is  used.  All  the  low-density  codes  used  in  this  study  were  chosen  so  that  no 
two  parity  check  sets  contain  more  than  one  digit  in  common,  thus  the 
independence  assumption  required  to  validate  the  first  iteration  is  satisfied. 

Zyablov  and  Pinsker  have  also  studied  low-density  codes.  They  obtained 
results  for  the  binary  erasure  channel  (BEC)  [9]  and  the  binary  symmetric 
channel  (BSC)  [10].  Both  results  concern  ensembles  of  codes.  For  the  BEC 
case,  their  ensemble  contains  codes  that  do  not  strictly  satisfy  the  definition 
of  low-density  codes  given  in  Section  1.1.  These  codes  have  a  parity  matrix 
with  a  fixed  number  of  l’s  in  each  column,  as  before,  but  the  number  of  l’s  in 
each  row  is  not  fixed. 

Zyablov  and  Pinsker  define  two  quantities,  Qq  and  u>0,  that  depend  on  the 
number  of  l’s  in  each  column  and  the  number  of  rows  in  the  parity  matrix. 
They  show  that  for  any  a  <  Oq  and  ui  <  uQ,  with  probability  P  approaching 
one  as  n  approaches  infinity,  with  other  code  parameters  fixed,  the  following 
event  occurs.  A  code  from  their  ensemble  has  minimum  distance  d  >um 
and  has  a  decoder  with  complexity  0(n  log  n)  that  can  correctly  decode  all 
erasures  with  multiplicity  <  an .  This  decoder  complexity  is  very  low  for  a 
block  code. 

A  decoder  that  achieves  this  result  works  by  identifying  the  parity  checks 
that  involve  exactly  one  erased  digit.  Each  of  these  erased  digits  has  only  one 
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value  that  will  satisfy  the  associated  parity  check,  thus  the  decoder  is  able  to 
fill  them  in  correctly.  The  process  repeats  until  all  erased  digits  are  filled. 

Zyablov  and  Pinsker  derive  an  analogous  result  for  the  BSC.  In  this 
case,  their  code  ensemble  is  equivalent  to  Gallager’s.  They  define  a  quantity 
o^5  that  depends  on  the  number  of  Ts  in  each  column  and  each  row  of  a 
code’s  parity  matrix.  For  any  a  <  5,  with  probability  P  approaching  one  as 

n  approaches  infinity,  with  other  code  parameters  fixed,  a  code  from  their 
ensemble  has  a  decoder  with  complexity  0(n  log  n)  that  can  decode  all  errors 
with  multiplicity  <  an . 

The  decoder  they  use  to  achieve  this  bound  starts  with  a  fixed  partition 
of  the  digits  into  q  —  j(k~  1)+1  subsets.  The  subsets  are  chosen  so  that  no 
two  digits  in  a  single  subset  are  linked,  in  the  sense  defined  earlier  of  occur¬ 
ring  together  in  a  common  parity  check.  Since  a  digit  can  be  linked  with  at 
most  j{k—  1)  other  digits,  such  a  partition  is  possible.  The  decoder  considers 
the  q  subsets  in  succession.  For  each  subset,  it  changes  all  the  digits  for 
which  less  than  half  the  j  parity  checks  in  which  they  occur  are  satisfied. 
The  process  repeats  until  at  least  half  the  parity  checks  involving  each  digit 
are  satisfied.  If  the  result  is  not  a  valid  codeword,  a  decoding  failure  is 
declared;  otherwise,  the  resulting  codeword  is  the  decoder  output. 
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CHAPTER  2 

AN  SDR  ALGORITHM  FOR  THE  BINARY  ERASURE  CHANNEL 

2.1  The  MNE  Ordering  Algorithm 

The  binary  erasure  channel  (BEC)  is  a  discrete,  memoryless  channel  with 
transition  probabilities  as  shown  in  Figure  1.1.  The  input  symbol  is  erased 
with  probability  e,  and  this  probability  is  independent  of  the  input.  If  not 
erased,  the  input  symbol  is  unchanged. 

Recall  that  an  SDR  algorithm  is  composed  of  two  parts  -  an  ordering 
algorithm,  that  orders  the  parity  checks,  and  a  sequential  decoder.  The  BEC 
is  well-suited  for  an  ordering  algorithm,  because  the  decoder  knows  exactly 
where  noise  occurs  in  the  message.  In  addition,  the  corrupted  digits  are 
equally  unreliable,  so  an  ordering  algorithm  does  not  have  to  compare  the 
effects  of  different  levels  of  noise.  Most  important,  there  exists  a  tight  upper 
bound  to  the  number  of  nodes  in  the  codeword  tree  visited  by  the  sequential 
decoder,  for  a  given  parity  check  ordering  and  erasure  pattern.  Here,  the 
sequential  decoder  “visits”  a  node  if  at  some  time  that  node  is  hypothesized 
to  be  on  the  path  corresponding  to  the  transmitted  codeword. 

This  upper  bound  is  found  by  counting  the  nodes  in  the  codeword  tree 
that  agree  with  the  unerased  portion  of  the  received  message.  (See  Figure 
2.1.)  Let  Ni  be  the  number  of  such  nodes  at  level  i  in  the  codeword  tree  for 
an  (n,  j ,1 c)  low-density  code.  Then 
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Figure  2.1.  An  example  illustrating  the  upper  bound  to  the  number  of  nodes 
visited  by  the  sequential  decoder.  Nodes  that  agree  with  the  received  message 
y  have  underlined  labels  and  darkened  edges.  This  is  the  same  codeword  tree 
fragment  shown  in  Figure  1.3. 
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Ni=2°‘Ni_l  1  <i<M 

where 

-  1  (2-2) 

M  =  n  j/k 

=  the  number  of  levels  in  the  codeword  tree 
ei  =  the  number  of  “new”  erasures  at  level  i 
=  the  number  of  erasures  in  the  n-set  at  level  i . 


The  terms  n-set  and  o-set  are  defined  in  Section  1.2.  The  quantity  <5-  is 
defined  as  follows.  We  assign  8.  —  1  if  e{  =  0  and  in  addition,  either:  1)  there 
are  no  erasures  in  the  o-set  at  level  i,  or  2)  the  parity  of  the  erasures  in  the  o- 
set  at  level  z,  taken  as  a  group,  assumes  only  one  value  among  the  surviving 
nodes  at  level  i.  Otherwise,  we  assign  ^  =  0.  In  particular,  <5(.  =  0  if  ei  >  1, 
and  <5-  =  1  if  the  parity  check  at  level  i  is  redundant,  given  the  preceding  parity 
checks. 

An  upper  bound  to  the  number  of  nodes  visited  by  the  sequential 
decoder  is  given  b} 

_  M  Mi 

M(y,P)  =  E  -  E  n  2a' .  (2.3) 

i-0  i-O  /-i 

where  y  is  the  received  message  and  P  is  the  parity  check  ordering. 


Proof  of  (2.3):  Clearly,  if  Ni  is  given  by  (2.1),  then  N  is  an  upper  bound 
to  the  number  of  nodes  visited  by  the  sequential  decoder.  To  see  why  (2.1)  is 

e  —  1 

valid,  first  suppose  et  >  1.  Then  there  are  2  '  ways  to  fill  the  ei  erasures  and 
satisfy  C(,  the  parity  check  at  level  i,  so  AT.  =  2 *'  ^Ni-V 

If  e-  =  0  and  there  are  no  erasures  in  the  o-set  at  level  i,  then  none  of  the 
digits  involved  in  C(-  were  erased.  This  means  there  is  exactly  one  way  to 
extend  each  of  the  surviving  nodes  at  level  i— 1,  so  iV(  =  N;_v 

Now  suppose  ei  =  0  and  condition  2)  for  6i  =  1  holds.  Then,  for  each  sur¬ 
viving  node  at  level  i— 1,  the  parity  of  the  erased  digits  in  the  o-set  at  level  i 
must  match  the  parity  obtained  when  these  erasures  are  filled  in  correctly. 
This  implies  Ni  = 

Finally,  if  ei  =  0  and  6-  =  0,  then  the  parity  of  the  erased  digits  in  the  o-set 
at  level  i  will  be  both  even  and  odd  among  different  surviving  nodes  at  level 
/— 1.  By  symmetry,  the  parity  for  half  these  nodes  will  match  the  parity 
obtained  when  the  erasures  are  filled  in  correctly.  Thus,  only  half  the  nodes 
will  be  extended  to  level  i,  so  iV-  =  Ni_l/  2.  This  establishes  (2.1),  and  there¬ 
fore  (2.3).  □ 

This  bound  is  tight  in  the  sense  that  for  a  given  erasure  pattern  and  parity 
check  ordering,  at  least  one  codeword  will,  when  transmitted,  require  the 
sequential  decoder  to  visit  N  nodes. 
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However,  it  is  not  practical  to  compute  N  because  it  is  difficult  to  deter¬ 
mine  when  <5-  =  1.  Even  if  one  could  compute  this  bound  easily,  there  does 
not  seem  to  be  an  efficient  way  to  minimize  this  bound  over  the  set  of  possi¬ 
ble  orderings. 

Instead,  one  can  use  a  greedy  heuristic  in  an  attempt  to  find  relatively 
good  orderings.  In  this  method,  the  ordering  is  obtained  by  choosing  the  par¬ 
ity  checks  one  at  a  time.  The  first  parity  check  is  chosen  to  minimize  Nv 
The  second  parity  check  is  chosen,  from  the  remaining  parity  checks,  to 
minimize  N and  so  on.  This  is  done  until  all  the  parity  checks  are  chosen, 
resulting  in  an  ordered  list.  In  general.  N.  depends  on  each  of  the  first  i  par¬ 
ity  checks.  However,  when  N-  is  minimized,  the  first  i—  1  parity  checks  have 
already  been  chosen,  so  Ni  is  not  minimized  over  the  set  of  all  orderings. 
Similarly,  N  is  not  necessarily  minimized.  However,  this  formulation  leads  to 
an  efficient  algorithm,  and  it  works  well  in  simulations. 

The  implemented  algorithm,  called  the  Minimum  New  Erasures  (MNE) 
algorithm,  differs  from  the  one  described  above  by  ignoring  the  <5.  term  in  the 
formula  for  ai  (2.2).  This  is  equivalent  to  choosing  C;  by  minimizing  et, 
hence  the  algorithm’s  name.  This  simplification  results  in  a  much  more  effi¬ 
cient  algorithm,  and  the  change  in  performance  is  minor.  To  see  this,  note 
that  6.  rarely  equals  one.  Furthermore,  ignoring  <5.  will  make  a  difference  only 
if  a  parity  check  with  <5(.  =  1  is  chosen  instead  of  a  parity  check  with  e.  =  0  and 
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6i  =  0.  In  any  case,  all  parity  checks  with  ei  —  0  will  be  chosen  before  any  with 
ei  >  1. 

A  straightforward  implementation  of  this  algorithm  searches  the  entire 
set  of  unused  parity  checks  every  time  a  new  parity  check  is  chosen.  Also, 
the  number  of  new  erasures  involved  in  each  parity  check  is  recalculated  for 
every  search,  since  the  definition  of  “new”  changes.  For  an  (n,  j ,k)  low  den- 
sity  code,  where  n  is  the  blocklength,  this  implementation  requires  0(j  n“/k) 
computation  and  O(jn)  memory.  A  more  efficient  implementation, 
presented  in  Section  A. 2,  requires  0{jn  )  computation  and  0(jn )  memory. 

2.2  Simulation  Results 

The  MNE  ordering  algorithm  was  implemented  on  a  computer.  To  form 
a  complete  decoder,  as  discussed  in  Section  1.1,  the  MNE  algorithm  was  cou¬ 
pled  with  the  stack  algorithm  (see  Section  7.2.7  of  Clark  and  Cain  [3]),  a  stan¬ 
dard  sequential  decoding  algorithm.  Although  the  sequential  decoding  algo¬ 
rithm  described  in  Section  3.3  was  used  with  the  algorithms  presented  in 
Chapter  3,  it  was  not  used  here,  because  the  stack  algorithm  was  found  to 
work  sufficiently  well  for  low-density  codes  on  the  BEC.  It  worked  well 
because  the  stack,  which  contains  only  codeword  fragments  that  agree  with 
the  unerased  portion  of  the  message,  stayed  small  -  less  than  30  entries  for 


each  trial. 


35 


Fixed  weight  pseudorandom  erasure  patterns  were  generated  and  decoded 
by  the  computer,  and  the  results  were  used  to  estimate  the  expected  amount 
of  computation  and  the  probability  of  decoding  failure.  The  measure  of  com¬ 
putation  was  the  number  of  steps  performed  by  the  sequential  decoder; 
specifically,  the  number  of  times  the  entry  at  the  top  of  the  stack  was 
extended.  The  all-zeros  codeword  was  always  used.  To  compensate  for  this, 
the  sequential  decoder  searched  the  codeword  tree  branches  in  reverse 
numerical  order,  and  thus  the  all-zeros  branch  was  always  searched  last.  As  a 
result,  the  expected  computation  and  probability  of  decoding  failure  are  upper 
bounds  to  what  would  be  expected  with  random  codewords. 

The  MNE  algorithm  was  tested  with  three  low-density  codes,  with  param¬ 
eters  (396,3,6),  (396,4,6),  and  (396,5,6).  All  three  codes  have  blocklength 
396,  and  their  designed  rates,  given  by  1 —j/k,  are  1/2,  1/3,  and  1/6,  respec¬ 
tively.  The  objectives  were  to  estimate  the  maximum  number  of  erasures  the 
decoding  algorithm  could  handle  and  to  compare  this  number  with  what  could 
theoretically  be  achieved  with  standard  sequential  decoding. 

Graphs  of  average  computation  and  probability  of  decoding  failure  versus 
number  of  erasures  are  shown  in  Figure  2.2.  To  gauge  the  effectiveness  of  the 
MNE  algorithm,  simulations  were  also  performed  with  a  random  ordering 
algorithm,  which  chooses  parity  check  orderings  with  equal  probability  for 
each  possible  ordering.  The  results  of  using  the  two  algorithms  with  the 
(396,5,6)  low-density  code  are  shown  in  Figure  2.3.  Finally,  to  extrapolate  the 
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Figure  2.3.  Simulation  results  for  random  ordering  and  the  MNE  algorithm, 
used  with  a  (396,5,6)  low-density  code. 
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results  of  Figure  2.2  to  codes  with  different  blocklengths,  simulations  were 
performed  using  codes  with  fixed  j  and  k  values,  but  varying  blocklengths. 
This  is  shown  in  Figure  2.4,  which  contains  graphs  of  normalized  computation 
versus  normalized  erasures  for  j  =  3,  k  =  6  and  /=  5,  k  =  6.  Here,  the  nor¬ 
malized  erasures  are  the  number  of  erasures  divided  by  n ,  and  the  normalized 
computation  is  the  amount  of  computation  divided  by  nj/k,  the  number  of 
levels  in  the  codeword  tree.  Note  that  all  codes  were  chosen  to  have  the  pro¬ 
perty  that  no  two  parity  check  sets  contain  more  than  one  digit  in  common, 
as  discussed  in  Section  1.4.  An  algorithm  to  generate  such  codes  is  included 
in  Section  A.l. 

To  illustrate  how  the  graphs  were  obtained,  Figure  2.5  contains  a  table  of 
simulation  results  for  the  MNE  algorithm  used  with  the  (396,5,6)  code.  The 
table  is  broken  up  into  two  parts  because  of  space  limitations;  two  sets  of  tri¬ 
als  were  not  performed  for  each  data  point.  After  each  trial,  which  consisted 
of  decoding  a  fixed  weight  psuedorandom  erasure  pattern,  the  average  compu¬ 
tation,  c,  and  the  measured  standard  deviation  of  computation,  s,  were  calcu¬ 
lated.  The  standard  deviation  of  c,  denoted  by  s,  was  estimated  by  s  =  s/V 7, 
where  t  is  the  number  of  trials.  This  relation  assumes  independent  trials. 
For  each  data  point,  the  computer  performed  at  least  200  trials  and  at  most 
3000;  within  this  range,  it  stopped  if  i  <  (.025)c.  The  number  of  trials  result¬ 
ing  in  decoding  errors,  N£,  and  the  number  of  aborted  trials,  NA,  were 
recorded  as  well.  A  trial  was  aborted  if  the  number  of  steps  performed  by 
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Figure  2.5.  Simulation  results  for  the  MNE  algorithm  used  with  a  (396,5,6) 
low-density  code. 
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the  sequential  decoder  reached  ten  thousand,  or  if  the  stack  size  reached  200. 
However,  the  limit  on  stack  size  was  unnecessary,  because  none  of  the  trials 
aborted  for  this  reason.  The  estimated  probability  of  decoding  failure  was 
given  by  pDF  =  (NE+NA)/T,  where  T  is  the  total  number  of  trials. 

Recall  that  one  objective  of  the  simulations  was  to  estimate  eMAX,  the 
maximum  number  of  erasures  that  could  feasibly  be  decoded  by  the  MNE 
algorithm.  From  the  results  shown  in  Figure  2.2,  one  can  estimate  eMAX  to  be 
approximately  165,  195,  and  210,  respectively,  for  the  (396,3,6),  (396,4,6),  and 
(396,5,6)  codes.  (See  Table  2.1.)  These  are  conservative  estimates;  they  are 
the  points  at  which  the  average  computation  first  shows  a  significant  increase 
from  its  minimum  possible  value,  n  j/k  +  1.  From  the  results  shown  in  Figure 
2.3,  the  maximum  number  of  erasures  for  random  ordering  with  the  (396,5,6) 
code  is  approximately  45.  As  stated  above,  the  performance  of  the  MNE  algo¬ 
rithm  is  significantly  better;  it  can  decode  roughly  4.7  times  as  many  erasures. 

Concerning  the  probability  of  decoding  failure,  note  that  decoding  errors 
occurred  much  less  often  than  decoding  failures.  A  decoding  error  occurs 
when  the  sequential  decoder  completes  its  search,  but  outputs  the  wrong 
codeword.  As  defined  here,  a  decoding  failure  occurs  if  a  trial  results  in  a 
decoding  error,  or  is  aborted  due  to  too  much  computation.  Out  of  23,609 
trials  used  to  generate  the  graphs  in  Figure  2.2,  only  4  resulted  in  decoding 
errors.  All  4  decoding  errors  occurred  using  the  weakest  code  -  the  (396,3,6) 
code.  Thus,  if  the  MNE  algorithm  is  used  on  a  channel  with  feedback  and 
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Table  2.1.  Comparison  of  the  MNE  algorithm  and  standard  sequential  decod¬ 
ing  for  the  binary  erasure  channel. 
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retransmission  capabilities,  one  can  achieve  decoding  error  rates  several  ord¬ 
ers  of  magnitude  smaller  than  pDF.  However,  the  effective  information  rate 
would  be  lower  than  the  code  rate,  due  to  retransmissions.  Note  that  for  the 
results  shown  in  Figure  2.4,  a  significant  number  of  decoding  errors  occurred 
when  using  the  (96,3,6)  and  (198,3,6)  codes.  It  seems  one  should  avoid  using 
blocklengths  this  small,  because  for  the  trials  used  to  generate  Figure  2.4, 
none  of  the  other  codes  had  any  decoding  errors. 

What  can  we  conclude  about  using  the  MNE  algorithm  at  rates  above  R0, 
the  computational  cutoff  rate  for  sequential  decoding?  Recall  the  estimated 
values  of  eMAX  for  the  (396,3,6),  (396,4,6),  and  (396,5,6)  codes;  they  are  listed 
in  Table  2.1.  These  values  represent  what  the  MNE  algorithm  can  achieve.  It 
is  well-known  that  the  formula  for  R0,  given  in  Section  1.1,  applied  to  the 
BEC  yields 

R0  =  l-lOg2(l+e),  (2.4) 

where  e  is  the  channel  erasure  probability  and  R0  is  measured  in  bits  per 
channel  use.  Inverting  this  formula  yields  the  maximum  erasure  probability 
that  sequential  decoding  can  handle  for  a  given  code  rate.  The  codes  listed 
above  have  designed  rates  of  1/2,  1/3,  and  1/6.  Corresponding  to  these  rates, 
the  maximum  values  of  e  for  sequential  decoding  are  listed  in  Table  2.1.  For 
each  code,  the  resulting  expected  number  of  erasures  per  codeword,  given  by 
ne  and  denoted  by  eSD,  is  listed  in  Table  2.1  as  well. 
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For  the  (396,4,6)  and  (396,5,6)  codes,  eMAX  for  the  MNE  algorithm  is 
significantly  less  than  eSD.  Thus,  it  seems  that  one  cannot  exceed  RQ  using 
the  MNE  algorithm  with  these  two  codes.  However,  the  (396,3,6)  code  looks 
more  promising.  The  value  of  eSD  for  this  code’s  rate  and  blocklength  is 
164.0,  while  eMAX  was  estimated  to  be  165.  These  values  are  too  close  to 
yield  a  clear  conclusion,  and  when  using  the  MNE  algorithm,  the  maximum 
acceptable  value  of  e  will  depend  on  the  specific  application.  Nevertheless,  it 
is  remarkable  that  a  randomly  chosen  (396,3,6)  low-density  code  performs  this 
well,  because  the  result  for  standard  sequential  decoding  is  an  upper  bound 
that  applies  to  all  tree  codes. 

Table  2.1  shows  that  the  MNE  algorithm  performs  better  with  respect  to 
sequential  decoding  as  the  code  rate  increases.  This  suggests  that  the  per¬ 
formance  may  continue  to  improve  with  code  rates  greater  than  1/2.  How¬ 
ever,  Gallager’s  result  discussed  in  Section  1.3  shows  that  the  typical 
minimum  distance  of  an  ensemble  of  (n,  j,k)  low-density  codes  grows  linearly 
in  n  only  if  ;>  3.  For  this  reason,  simulations  were  not  performed  with 
(3%, 2, 6)  or  (3%,  1,6)  codes.  It  would  be  desirable  to  study  low-density  codes 
with  rate  R=l  —  j/k  >  1/2  and  ;>3  to  determine  whether  the  MNE  algo¬ 
rithm  can  clearly  outperform  standard  sequential  decoding. 

To  achieve  an  arbitrarily  low  probability  of  decoding  error,  one  must  use 
codes  with  arbitrarily  large  blocklengths.  This  leads  one  to  question  whether 
the  results  presented  above  can  be  extrapolated  to  low-density  codes  with 
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larger  blocklengths.  The  graphs  shown  in  Figure  2.4  shed  some  light  on  this. 
Both  graphs  seem  to  indicate  that  the  cutoff  point,  where  the  average  compu¬ 
tation  starts  to  increase,  occurs  at  roughly  the  same  value  of  normalized  eras¬ 
ures,  as  n  varies  and  with  /  and  k  fixed.  In  other  words,  for  fixed  j  and  k, 
the  number  of  decodable  erasures  when  using  the  MNE  algorithm  increases 
linearly  in  n.  If  this  hypothesis  is  true,  then  using  the  MNE  algorithm  with 
low-density  codes  is  truly  comparable  to  standard  sequential  decoding,  in  the 
sense  that  codes  with  arbitrarily  long  blocklengths  will  work  well  on  a  channel 
with  fixed  erasure  probability.  This  behavior  is  expected,  since  the  typical 
minimum  distance  of  an  ensemble  of  (n,  j ,k)  low-density  codes  grows  linearly 
in  rt ,  for  j  >  3. 

Note  that  this  discussion  implicitly  assumes  that  the  codes  used  in  the 
simulations  are  “typical”  low-density  codes.  This  assumption  is  reasonable  in 
light  of  Gallager’s  result  presented  in  Section  1.3.  It  states  that  almost  all  the 
low-density  codes  in  his  ensemble  have  minimum  distance  greater  than  a  sin¬ 
gle  lower  bound,  In  addition,  in  Chapter  3,  the  simulation  results 

obtained  using  two  randomly  chosen  (396,5,6)  low-density  codes  are  found  to 
be  very  close. 

One  feature  that  limits  the  SDR  approach,  that  is  not  encountered  in 
standard  sequential  decoding,  is  that  codeword  trees  for  low-density  jodes 
tend  to  grow  more  rapidly  near  the  beginning  than  near  the  end.  This  hap¬ 
pens  regardless  of  the  parity  check  ordering,  as  discussed  in  Section  1.2. 
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Thus,  the  early  part  of  the  tree  looks  like  a  code  with  higher  rate  than  the  ori¬ 
ginal  code.  This  is  a  drawback,  since  the  sequential  decoder  works  well  only 
at  rates  less  than  R0.  However,  the  MNE  algorithm  tends  to  push  erased 
digits  toward  the  end  of  the  tree,  and  this  helps  the  sequential  decoder.  The 
net  result  of  these  effects  determines  whether  the  overall  performance  is 
better  than  standard  sequential  decoding.  Another  limitation  of  the  SDR 
approach  is  that  the  distance  properties  of  low-density  codes  are  probably  not 
as  good  as  those  of  arbitrary  tree  codes. 
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CHAPTER  3 

SDR  ALGORITHMS  FOR  THE  BINARY  SYMMETRIC  CHANNEL 

3.1  Codeword  Tree  Ordering  Algorithms 

The  binary  symmetric  channel  (BSC)  is  a  discrete,  memoryless  channel 
with  transition  probabilities  as  shown  in  Figure  3.1.  With  probability  p,  an 
error  occurs;  otherwise,  the  input  symbol  is  unchanged.  The  error  vector,  e, 
is  defined  by 

ei  -  *,-©?,•  >  (3.1) 

where  @  denotes  mod  2  addition.  In  contrast  with  the  binary  erasure  chan¬ 
nel,  it  is  not  immediately  apparent  when  an  input  symbol  is  incorrectly 
received. 

Since  every  digit  in  the  received  message,  taken  by  itself,  is  equally  reli¬ 
able,  it  would  seem  that  the  ordering  algorithm  has  no  information  to  work 
with.  With  this  view  in  mind,  the  algorithm  would  choose  an  ordering  that  in 
some  way  minimized  the  size  of  the  codeword  tree,  and  it  would  use  this  ord¬ 
ering  regardless  of  the  received  message. 

However,  one  may  generate  a  reliability  measure  for  a  received  digit  by 
considering  the  values  of  other  digits  and  using  the  structure  of  the  code.  For 
example,  suppose  one  of  the  parity  equations  of  the  code  states  that  for  every 
codeword,  x1  and  x5  must  sum  to  zero,  mod  2.  If  y :  and  v5  are  both  zero, 
then  the  probability  that  yj  is  in  error  is  less  than  its  a  priori  probability,  p, 


assuming  p< 0.5  and  equally  probable  codewords.  With  this  approach,  it  is 
possible  to  obtain  different  reliability  levels  for  different  digits,  and  this  infor¬ 
mation  can  be  used  by  the  ordering  algorithm  to  generate  an  ordering  that 
depends  on  the  received  message. 

One  may  think  of  a  reliability  measure  for  bit  i  as  an  estimate  of  1— er 
In  other  words,  the  optimal  reliability  measure,  if  it  could  be  computed, 
would  equal  one  if  y  •  =  xi ,  and  zero  otherwise.  Since  ei  is  modeled  as  a  ran¬ 
dom  variable,  a  Bayesian  estimator  can  be  used.  One  cost  function  for  com¬ 
paring  Bayesian  estimators  is  the  mean-squared  error  ,  E($—  6)2,  where  $  is 
the  estimate  and  9  is  the  quantity  being  estimated.  Given  that  we  observe  y, 
the  channel  output,  the  minimum  mean-squared  error  (MMSE)  estimate  is  the 
conditional  mean,  E(l— ei  |y).  Equivalently,  this  estimate  is  given  by 

P(^=^|y)* 

However,  it  is  not  advisable  to  use  this  estimate  in  this  situation.  Condi¬ 
tioning  on  the  entire  received  message  would  probably  require  a  great  deal  of 
computation,  and  the  ordering  algorithm’s  purpose  is  to  reduce  the  complexity 
of  the  sequential  decoder.  In  addition,  computing  this  estimate  corresponds, 
in  a  sense,  to  fully  decoding  the  received  message.  The  output,  x,  would  be 
given  by 

=  arg  max  P(x.  =  j  |  y).  (3.2) 

/'-O.l 


Instead  of  being  the  most  probable  codeword,  x  would  minimize  the 
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probability  of  error  for  each  digit.  Such  a  scheme,  sometimes  referred  to  as 
probabilistic  decoding,  does  not  necessarily  output  a  codeword.  This 
approach  would  invalidate  the  proposed  structure  of  the  decoding  algorithm, 
because  the  ordering  algorithm  is  intended  only  as  a  preprocessor  for  the 
sequential  decoder. 

Instead,  one  could  consider  using  a  coarse,  easily  computable  reliability 
measure  that  would  nevertheless  lead  to  an  improvement  over  the  static  order¬ 
ing  described  above.  Fortunately,  one  may  compute  such  a  measure  for  low- 
density  codes.  For  a  given  digit  i,  it  is  found  by  considering  Vt,  the  number 
of  violated  parity  checks  involving  digit  i.  The  reliability  measure  is  defined 
by 


(3.3) 


where  the  joint  probability  distribution  of  the  x(’s  is  obtained  by  assuming 
each  codeword  is  transmitted  with  equal  probability.  This  quantity  may  be 
interpreted  as  the  MMSE  estimate  of  1— e-,  given  that  we  observe  only  V  . 
Assuming  that  no  two  parity  checks  have  more  than  one  digit  in  common,  its 
value  is  given  by  the  following  expression: 


PC*-*/ TO 


_ 1 _ 

i  +  p/?;~2Vy(i-p) 


(3.4) 


where 
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1—  (1— 2p)*-1 

8=  - 1 - — - 

l+(l-2p)*_1 

p  =  channel  error  probability 
j  ,k  =  low— density  code  parameters 


A  proof  is  given  below,  but  first  note  that  in  an  ( n,j,k )  low-density  code, 
each  digit  appears  in  j  parity  checks,  thus 

0  <V,<j.  (3.5) 

For  a  given  channel  and  low-density  code,  P(yi  —  xi  |  Vi )  depends  only  on  Vt, 
hence  the  reliability  measure  will  assume  only  j+ 1  distinct  values.  These 
values  may  be  efficiently  stored  in  a  table,  so  that  one  need  not  recalculate 
the  above  expression. 

Proof  of  (3.4):  Let  Ci ,  be  one  of  the  j  parity  checks  involving  digit  i, 
the  event  that  C- ,  is  not  satisfied  by  y,  and 


a  =  P(A[\yi  =  xi).  (3.6) 

Parity  check  C .  .  is  not  satisfied  if  and  only  if  there  are  an  odd  number  of 
*  >* 

errors  among  the  digits  involved  in  C.  r  There  are  k-~  1  digits  other  than  i 
involved  in  C; ,,  and  Lemma  4.1  of  [7]  shows  that 


l  _  1  +  (1-2P)*"1 
2 

1  -  (1-2 p)k'x 
2 


(3.7) 


52 


Whether  or  not  a  parity  check  is  satisfied  depends  only  on  the  error  vec¬ 
tor,  e.  Recall  the  assumption  that  no  two  parity  checks  have  more  than  one 
digit  in  common.  Then,  given  yi  =  xi ,  the  j  events  A  p  •  •  •  ,  Aj  depend  on 
mutually  exclusive  subsets  of  e,  and  are  therefore  independent.  Then  Vi  is 
binomially  distributed  with  parameters  ( j,a ),  and 

PC^-H?,--*,-)  -  [}y  )a(1-a);_v.  (3.8) 

Given  y^x,,  At  occurs  if  and  only  if  there  are  an  even  number  of  errors 
over  the  remaining  k—1  digits  in  Ci  Thus, 

P  (A,\yi^xj)  =  \-cl.  (3.9) 

The  random  variable  Vi  is  again  binomially  distributed,  but  now  with  parame¬ 
ters  (/',  1— a). 

nVi-v^x,)  =  (  '  )  (l-a)V-v.  (3.10) 

The  desired  probability,  P(y4-  =  xi  j  Vt),  may  be  written  as 

P(V'i=v|y|.  =  .v.)P(y/  =  x.) 

P (7,  =  x.t  |  V.  =  v)  = - . 

P( Vi-v\ y.  =  Xi )  P )  +  P( Vt  =  v | y i  +  Xi )  P(y , * ) 

(3.11) 

Substituting  (3.8)  and  (3.10)  into  (3.11),  and  defining  =  a/(  1— o),  we  obtain 
(3.4).  □ 


53 


This  reliability  measure  is  closely  related  to  one  of  Gallager’s  decoding 
algorithms  for  low-density  codes  (pp.  42-45  of  Gallager  [7]).  A  brief  descrip¬ 
tion  of  the  algorithm  is  given  here;  it  is  described  more  fully  in  Section  1.4. 
Each  digit  is  associated  with  /  values  between  zero  and  one,  and  these  values 
are  interpreted  as  the  probability  that  the  transmitted  symbol  was  a  one,  con¬ 
ditioned  on  different  subsets  of  the  received  message.  The  algorithm  is  itera¬ 
tive.  Initially,  each  probability  is  conditioned  on  the  received  value  of  a  single 
digit,  and  in  each  iteration  the  conditioning  sets  are  expanded  to  include  more 
of  the  received  message.  If  the  decoder  is  successful,  all  the  values  associated 
with  each  digit  converge  either  to  zero  or  to  one.  The  independence  assump¬ 
tions  used  to  derive  the  update  equations  are  violated  after  a  relatively  small 
number  of  iterations,  but  the  procedure  is  continued  anyway  and  is  found  to 
yield  a  good  decoding  algorithm. 

The  results  of  the  first  iteration  of  Gallager’s  algorithm  are  very  similar  to 
the  reliability  measure  defined  above.  Let  i  be  a  given  digit.  In  the  first  itera¬ 
tion,  the  conditioning  sets  for  i  are  expanded  to  include  5{-,  the  set  of  all  digits 
that  occur  in  the  j  parity  checks  containing  i.  However,  to  validate  the  next 
iteration,  each  of  the  /  probability  values  associated  with  i  ignores  the  digits 
involved  in  one  of  the  parity  checks.  If,  instead,  we  calculate  a  single  value 
that  considers  all  the  digits  in  S-,  then  we  obtain  the  same  quantity  as  the  reli¬ 
ability  measure  defined  above,  under  the  conditions  defined  below. 
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Recall  that  our  reliability  measure  is  not  defined  by  conditioning  on  St, 
but  on  V;,  the  number  of  violated  parity  checks  containing  i.  Even  though  Vi 
is  a  coarser  statistic  than  5i?  the  two  definitions  result  in  the  same  value  if  the 
following  two  conditions  hold: 

(1)  Each  codeword  in  the  code  is  transmitted  with  equal  probability. 

(2)  There  is  no  derived  parity  equation,  involving  only  digits  in  S-,  that  is 
satisfied  by  the  code  and  linearly  independent  of  the  j  equations  involv¬ 
ing  digit  i . 

This  is  proved  as  Proposition  3.1  below.  We  use  V.  instead  of  5.  because 
(3.4)  is  easier  to  evaluate,  though  less  general,  since  it  applies  only  to  the 
BSC,  than  the  formula  for  P(y;.  =  xi\Si)  given  by  Gallager.  For  comparison, 
P (yi~xi  |S;)  is  given  by  any  one  of  Equations  (4.1)  or  (4.6)  of  [7],  or  Equation 
(1.14)  in  this  thesis.  To  show  that  P(y;  =  xi  \  Vt)  —  P^.  =  xt  |5(),  we  can  obtain 
(3.4)  directly  from  Gallager’s  formula  for  ?(yi  —  xi  |St),  by  inserting  values 
appropriate  for  the  BSC.  However,  Gallager’s  probability  model  is  defined 
differently  than  the  one  used  here.  Nevertheless,  we  can  show  that  the 
models  are  equivalent  when  we  restrict  our  attention  to  the  digits  in  5-,  which 
is  sufficient  for  our  purposes.  Instead,  we  will  show 
P(y;  =  X;  IV])  =  P(yi  =  xi  J  5- )  for  the  model  used  here  -  namely,  assumptions 
(1)  and  (2)  above,  together  with  a  memoryless  BSC. 
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Note  that  an  equation  as  described  in  (2)  can  occur  because  the  code¬ 
words  satisfy  not  only  the  n  j/k  parity  checks  in  the  low-density  code’s  defini¬ 
tion,  but  all  linear  combinations  of  them  as  well.  However,  as  Gallager 
states,  the  effect  of  the  extra  dependencies  that  occur  if  condition  (2)  is 
violated  is  typically  small. 

We  use  the  following  fact.  Its  proof  is  elementary  and  is  omitted  here. 

Fact.  Let  C  be  a  linear  code  with  blocklength  n,  and  let  R  C  {1,2 
Let  PC(R )  be  the  set  of  all  possible  parity  check  equations  that  involve  only 
digits  in  R  and  that  are  satisfied  by  all  codewords  in  C,  and  let  xR  be  an 
assignment  of  values  to  digits  in  R.  Then  xR  will  match  some  codeword  in  C 
if  and  only  if  xR  satisfies  PC(R).  In  addition,  if  a  random  codeword  is  chosen 
uniformly  from  C,  then  every  valid  xR  is  equally  probable. 

Proposition  3.1.  P(y,-  =  xi  [  V^)  =  P(y-  =  xt  |S(),  under  conditions  (1)  and 
(2)  stated  above. 

Proof:  Let  R  C  {1,2 be  the  union  of  the  j  parity  check  sets  involv¬ 
ing  digit  i.  In  other  words,  R  is  the  set  of  indices  of  the  digits  that  make  up 
Sr  For  all  vectors  in  this  proof,  we  restrict  our  attention  to  elements  indexed 
by  R.  This  applies  to  codewords  as  well  as  error  vectors. 

Define  v(a)  for  a  vector  a  £  {0,1}  to  be  the  number  of  violated  parity 
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checks  involving  digit  i.  Suppose  at  and  a-,  satisfy  v(ax)=  v(a2).  Then  we  will 
show 

P(y.=xi\Si=sil)=  P(yi=xi  |S(=a2)  .  (3.12) 

Let  Tx—  {(x^,e^),(xp\ep^),...}  be  the  set  of  all  pairs  of  codewords  and 
error  vectors  that  result  in  a  received  message  with  5-=  ap  and  similarly 
define  T2=  {(x^.e^),^2^^),...}  for  a2.  We  will  demonstrate  a  one-to- 
one  mapping  from  Tl  onto  T2,  and  use  this  to  show  P(Tl)=  P{T2).  The 
underlying  probability  model  is  that  the  transmitted  codeword  is  chosen  uni¬ 
formly  from  the  set  of  all  codewords,  and  the  error  vector  is  generated  by  a 
memoryless  BSC. 

Note  that  every  valid  codeword  occurs  in  both  Tx  and  T2.  Recall  that  we 
consider  only  digits  with  indices  in  R.  From  the  assumptions  and  the  Fact,  a 
valid  codeword  is  a  vector  in  {0,1}  that  satisfies  the  j  parity  checks  that 
involve  digit  i.  In  addition,  each  such  vector  is  equally  probable. 

Without  loss  of  generality,  we  can  assume  that  the  violated  parity  checks 
that  contribute  to  v(aj)  are  the  same  parity  checks  that  contribute  to  v(a2). 
This  implies  that  x+aj-l^  is  a  valid  codeword  if  x  is.  Otherwise,  we  can 
achieve  this  condition  by  rearranging  the  digits  in  each  of  the  vectors  in  T2, 
creating  a  new  set  T2.  Specifically,  the  digits  involved  in  some  violated  parity 
checks  are  interchanged  with  those  involved  in  some  satisfied  checks.  The 
codewords  are  still  valid,  and  clearly  P(T2)=  P(T .,). 


The  desired  mapping  <f>  from  Tj  to  T2  is  given  by 

<^((x,e))  =  (x+a1+a2,e) .  (3.13) 

It  is  clearly  one-to-one  and  onto.  Since  each  codeword  is  equally  probable, 
this  shows  P(T])=P(T2).  Equivalently,  since  P{T^)  =  /’(S^aj)  and 
P(T~)=P{Si= a2),  we  have  P(5i=a1)  =  P(S/=a2).  Similarly,  we  can  show 
P(yi—xi  and  5;-= a2)  =P(yi=xi  and  S(=a 2).  In  this  case,  the  error  vectors  in 
T j  and  T2  are  constrained  to  be  zero  in  position  i.  Therefore, 

P(y,=^  I =  P(yi=xi  1 5-=a2) .  (3. 14) 

Finally,  since  P(y(  =  xi  \  V,)  is  a  weighted  average  of  P{y  =xi  |5-=a)  for  all  a 
with  v  (a  )=  V. ,  this  implies  P(y <.  =  xt  |  V{ )  •  P(y .  =  xi  \  ) .  □ 

Once  we  have  an  easily  computable  reliability  measure,  we  must  decide 
how  to  incorporate  it  in  an  ordering  algorithm.  This  question  may  be 
addressed  by  considering  the  ordering  algorithm  for  the  binary  erasure  chan¬ 
nel  (BEC).  To  recall,  the  BEC  algorithm  does  not  compare  complete  f  ity 
check  orderings;  instead,  it  constructs  a  single  ordering  by  choosing  parity 
checks  one  at  a  time.  At  each  stage,  the  chosen  parity  check  is  one  that 
minimizes  the  value  of  a  certain  function  (the  objective  function),  and  this 
function  is  derived  from  an  upper  bound  to  the  number  of  nodes  visited  by 
the  sequential  decoder. 
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This  approach  is  not  directly  applicable  to  the  BSC;  unlike  the  case  with 
the  BEC,  we  have  not  found  a  reasonably  tight  bound  on  the  number  of 
nodes  visited  by  the  sequential  decoder,  given  the  received  message  and  the 
parity  check  ordering.  It  seems  that  such  a  bound  would  have  to  take  into 
account  the  fine  structure  of  the  low-density  code  being  used  (to  a  greater 
extent  than  the  BEC  bound,  which  only  considered  the  overlap  between  a 
given  parity  check  and  the  union  of  the  previously  chosen  parity  checks). 

Nevertheless,  one  can  obtain  an  effective  algorithm  by  choosing  an 
appropriate  objective  function.  Specifically,  we  consider  objective  functions 
that  tend  to  maximize  the  reliabilities  of  the  new  digits  and  tend  to  minimize 
the  number  of  new  digits,  where  the  new  digits  in  a  parity  check  are  the  ones 
not  contained  in  any  of  the  previously  chosen  parity  checks.  Recall  that  if  a 
given  parity  check  is  chosen  to  be  ith  in  the  final  ordering,  then  the  new  digits 
in  this  parity  check  are  the  ones  filled  in  at  level  i  in  the  codeword  tree. 

Consider  the  effect  of  this  algorithm  on  a  given  level  of  the  codeword 
tree.  If  we  maximize  the  reliability  of  the  new  digits,  then  the  sequential 
decoder  will  be  less  likely  to  start  along  an  incorrect  path,  or  if  it  is  already 
on  an  incorrect  path,  it  will  be  more  likely  to  detect  this.  On  the  other  hand, 
when  we  minimize  the  number  of  new  digits,  the  growth  of  the  tree  at  this 
level  will  also  be  minimized.  As  a  result,  there  are  fewer  nodes  to  search, 
and  there  is  less  likely  to  be  an  outgoing  branch  close  enough  to  the  correct 
branch  to  appear  correct.  However,  it  may  not  be  possible  to  fulfill  these  two 
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aims  simultaneously,  thus  the  objective  function  will  have  to  define  their  rela¬ 
tive  importance. 

Since  the  algorithm  avoids  choosing  parity  checks  with  unreliable  digits, 
the  overall  effect  is  to  push  noisy  digits  deeper  into  the  codeword  tree.  To 
see  why  this  is  a  good  thing  to  do,  recall  that  regardless  of  the  parity  check 
ordering,  the  codeword  tree  tends  to  grow  more  slowly  at  deeper  levels.  (See 
Section  1.4.)  For  this  reason,  a  wrong  move  by  the  sequential  decoder  at  the 
beginning  of  the  tree  will  probably  require  more  computation  to  resolve  than  a 
wrong  move  toward  the  end.  In  addition,  errors  toward  the  end  of  the  tree 
are  less  likely  to  cause  a  wrong  move,  especially  if  the  error  occurs  at  a  level 
where  the  tree  does  not  grow  in  size  (one  new  digit  per  level). 

These  objective  functions  do  not  depend  on  the  reliabilities  of  the  old 
(i.e.,  already  filled)  digits,  but  there  are  situations  where  such  dependence 
may  be  desirable.  For  example,  one  could  choose  successive  parity  checks  to 
minimize  the  reliabilities  of  the  old  digits  in  the  new  parity  checks  in  order  to 
detect  incorrect  moves  by  the  sequential  decoder  sooner,  since  an  incorrect 
assignment  can  be  detected  only  at  a  level  where  it  occurs  as  an  old  digit. 
However,  it  would  be  difficult  to  tell  when  this  would  be  helpful,  since  an 
incorrect  assignment  of  an  old  digit  may  be  interpreted  as  a  transmission  error 
in  a  new  digit.  In  any  case,  it  seems  that  this  would  rarely  be  more  important 


than  the  factors  discussed  above. 
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Several  different  objective  functions  with  the  properties  described  above 
were  used  in  algorithms  implemented  on  a  computer.  They  are  listed  in  Fig¬ 
ure  3.2.  A  description  of  these  functions  is  given  below,  and  simulation 
results  are  presented  in  Section  3.2.  First,  we  present  some  definitions,  and 
discuss  the  computational  complexity  of  the  algorithms.  We  use  the  following 
terminology.  The  objective  function  is  denoted  by  E,  and  its  domain  is  the 
set  of  parity  checks.  For  a  given  parity  check,  N  is  the  set  of  new  digits, 
which  are  defined  as  above. 

Each  ordering  algorithm  operates  as  follows.  The  objective  function  is 
evaluated  at  each  parity  check,  and  a  parity  check  that  minimizes  the  function 
is  chosen.  Since  N  will  change  for  some  of  the  parity  checks,  E  is  re¬ 
evaluated  for  each  remaining  parity  check  before  the  next  choice  is  made. 
This  procedure  is  repeated  until  a  complete  ordering  is  obtained. 

The  approach  described  above  requires  0(j~n~/k)  computation  and 
0{jn)  memory.  A  more  efficient  implementation  is  possible  if  E  assumes 
only  a  finite  set  of  values,  and  has  the  additional  property  that  when  a  parity 
check’s  E  value  changes,  it  cannot  return  to  ?  previously  held  value.  These 
conditions  are  saTsfied  by  discretized  versions  of  the  objective  functions 
described  below,  except  for  functions  7.1  and  7.2.  With  these  conditions,  an 
implementation  exists  with  0(n j  +  n jV/k)  computation  and  0(n j  +  n jV/k) 
memory,  where  V  is  the  number  of  values  £  can  assume.  The  implementa¬ 
tion  is  similar  to  the  0(n )  version  of  the  MNE  algorithm  presente-.  in  Section 
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1.  E=£(aV, +,.;?) 

ieN 

1.1  u  =  l.  ..'5=0  1.3  a=0,  .5=1 

1.2  a=l,  0 <  ,5<  l//c  1.4  0<  cv<  1/ jk ,  3=  1 


2.  £  = 


EE/I^I 

ieN 

0 


if  |N|  >  1, 


otherwise. 


3.  £  =  >  h(x)  =  —  v logv  —  ( 1  —x ) log ( l—x ) 

ieN 


4.  £  =  fl  [l+2V^(l-£) 

ieN 


5.1  £  =2|w|"1£min(/^.l-^) 

i  eN 


5.2  £  =  £min(pf,l-p.) 

ieN 


6.1  £  =21N|_1[l-nmax(/vl-£,)] 

ieN 


6.2  £  =  1— n  max^.l-/;,) 

ieN 


7.1 


£ 


[acmo/  (/ )— desired  (/ )  ]' 


o  </</, 

total(l)*0 


(total  (/))* 


7.2 


E 


|  actual  (l )—  desired  (/ )  | 


o  </<;. 

total  (/)#0 


total  (l) 


Figure  3.2.  List  of  objective  functions. 
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A. 2.  The  main  difference  from  the  MNE  algorithm  concerns  the  value  of 
each  digit’s  contribution  to  E.  With  the  MNE  algorithm,  E  equals  the 
number  of  erasures  in  N.  Thus,  a  digit’s  E  contribution  is  one  or  zero, 
depending  on  whether  or  not  the  digit  is  an  erasure.  With  the  BSC  objective 
functions,  we  must  store  each  digit’s  E  contribution  in  a  size  n  array.  Using 
this  array,  we  can  determine  the  effect  on  E  of  removing  a  digit  from  A  in  a 
constant  number  of  steps.  Without  the  array,  we  must  recalculate  E  from 
scratch,  which  requires  0(k )  steps. 

3.1.1  Objective  functions 

Definitions.  As  above,  E  is  the  objective  function,  and  N  is  the  set  of 
new  digits  for  a  parity  check.  For  1  <i  <  n,  we  define 

Pi-P(y,+ ^  iv().  (3.15) 

The  function  PQ  is  a  probability  distribution  on  x  and  y  that  differs  from  the 
BSC  model  by  setting  P0(y,  #  xt)  —  pr 

1.  E  -  £  (aV.+P)  , 

i  €  N 

where  a  and  /3  are  constants,  and  V.  is  the  number  of  violated  parity  checks 
involving  yr  Using  V{  itself  as  a  quantitative  measure  of  uncertainty  was  arbi¬ 
trary,  but  this  objective  function  was  found  to  work  well.  Different  values  of 
a  and  /?  yield  different  orderings,  and  the  following  values  were  implemented. 
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1.1  <2=1,  /?= 0 

Minimizes  V, . 

/e  zv 

1.2  a=l,  Q<(3<  1/k 

Since  each  V.  is  an  integer,  and  |iV|</c,  this  causes  the  algorithm  to  minim- 

ize  Y,  Vi »  ancl  break  ties  by  minimizing  \N\. 
ieN 

1.3  a=0,  0=1 

This  yields  a  static  ordering,  which  minimizes  |  N  \ .  The  ordering  is  indepen¬ 
dent  of  the  received  vector  y.  It  is  obtained  without  any  knowledge  of  the 
reliabilities  of  the  individual  digits. 

1.4  0<a<l/;'A;,  0=1 

Since  V.  <;,  Thus,  the  algorithm  minimizes  |iV|  and  breaks  ties 

ieN 

by  minimizing  V-. 

<6  ZV 


2. 


E  =  1 


EK/I^I 

»6  ZV 


if  |/V|>  1, 


0  otherwise. 


This  algorithm  tries  to  choose  a  parity  check  with  smallest  uncertainty  per 
new  digit. 
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3.  E  =  £  A  (ft), 
ie  w 

where  /z  is  the  entropy  function, 


/z(.v)  =  —  a: log.*—  (1— Ar)log(l — x)  . 


4.  £  =  n  [l+2Vft(l -/>,)  J 

ieN 

One  interpretation  for  this  algorithm  is  that  it  maximizes  (/?0(y?.)  —  r.), 

ie  W 

where 


(P,)=  l-log2[l+2Vp,(l-pi)  ] 

=  the  computational  cutoff  rate  in  bits  for 
a  BSC  with  crossover  probability  pt 


and 


r,  =  (|A/|-l)/|iV| 

=  the  approximate  local  code  rate  in  bits. 

Motivation  for  this  objective  function  comes  from  two  bounds  to  the  expected 
amount  of  computation  performed  by  a  sequential  decoder  -  an  upper  bound 
due  to  Gallager  (p.  279  of  [5])  and  a  lower  bound  due  to  Arikan  (Lemmas  3. 1 
and  6.1  of  [1]).  If  one  extends  these  bounds  to  the  case  where  the  code  rate 
and  the  crossover  probability  can  vary  with  time,  the  given  objective  function 
minimizes  both  these  bounds.  However,  the  original  bounds  are  loose,  and 
the  extensions  may  not  be  valid. 
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5.1  E=2|w|-1£min(p,.,l-pi) 

ie  N 

5.2  E  =  £  min(/>.,l-p.) 

ie  n 

The  expression  Yj  mi nCp^l— p2)  is  a  union  bound  approximation  to 
ie  n 

?0(Xi*xi  for  some  i£  N),  where  P0  is  the  probability  distribution  defined 
previously,  x(  =  arg  max  T(x,yi),  and  T  is  the  sequential  decoder  metric. 

X— 0,1 

Since  r^xj.,^)  >  r(l—^i,yi),  the  sequential  decoder  will  most  likely  assume 
x  •  =  x ■  on  its  first  pass.  The  quantity  2^-1  is  a  cost  chosen  to  reflect  the 
local  growth  of  the  codeword  tree  if  the  current  parity  check  were  chosen. 


6.1  E-2|w|-1[l-nmax(pi,l-p1)] 

ie  n 

6.2  E  =  [1-  n  max(p.,l-p.)] 

•  ew 

The  expression  [1—  max(/?.,l— p.)]  equals  P0(x- ^ xi  for  some /€  AO*  As 

ie  N 

with  objective  function  5.1,  2^_1  is  a  cost  associated  with  this  event. 


7.1 


7.2 


E 

o  </<;, 

total  (1)^0 


A 

[actual  (/)— desired  (/)]" 
(total  (l))2 


E 

o</</, 

total  (1)^0 


|  actual  (l )  -desired  (l )  | 
total  (l) 


Here,  total(l)  is  the  total  number  of  digits  in  the  received  message  with 
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V.  =  /.  The  quantity  desired(l )  is  defined  to  be  (nQ/n) total (/),  where  n0  is 
the  number  of  digits  encountered  in  the  codeword  tree  so  far,  and  actual (l)  is 
the  number  of  such  digits  with  Vj  =  l.  Thus,  these  objective  functions  try  to 
distribute  the  digits  with  a  given  value  of  Vi  evenly  throughout  the  tree. 

3.2  Simulation  Results 

The  objective  functions  described  in  Section  3.1  were  used  in  codeword 
tree  ordering  algorithms  implemented  on  a  computer.  To  form  a  complete 
decoder,  as  discussed  in  Section  1.1,  each  ordering  algorithm  was  coupled 
with  a  sequential  decoder.  Because  standard  sequential  decoding  algorithms 
did  not  work  well  with  low-density  codes  on  the  BSC,  a  new  sequential 
decoder  was  used.  It  is  described  in  Section  3.3. 

The  simulations  performed  for  the  BSC  algorithms  are  similar  to  those 
described  in  Section  2.2.  Fixed  weight  ps-uedorandom  error  vectors  were  gen¬ 
erated  and  decoded  by  a  computer,  and  the  results  were  used  to  estimate  the 
expected  amount  of  computation  and  the  probability  of  decoding  failure  for 
each  algorithm.  In  this  case,  the  measure  of  computation  was  the  number  of 
forward  and  lateral  moves  performed  by  the  sequential  decoder.  As  before, 
the  all-zeros  codeword  was  always  used.  To  compensate  for  this,  the  sequen¬ 
tial  decoder  searched  the  codeword  tree  branches  in  reverse  numerical  order, 
thus  the  all-zeros  branch  was  always  searched  last.  However,  because  of  the 
sequential  decoder  used,  the  expected  computation  and  probability  of 
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decoding  failure  are  not  necessarily  upper  bounds  to  what  would  be  obtained 
with  random  codewords,  as  was  the  case  with  the  BEC  results. 

The  objective  functions  were  compared  using  a  fixed  (396,5,6)  low- 
density  code,  denoted  by  c39656.  Simulations  were  performed  to  estimate  the 
expected  computation  and  probability  of  decoding  failure  as  a  function  of  the 
number  of  channel  errors.  To  study  the  effects  of  different  j  and  k  values, 
objective  functions  1.2  and  3  were  tested  with  (396,3,6)  and  (400,4,8)  low- 
density  codes  as  well.  To  gauge  the  effectiveness  of  the  algorithms,  we  imple¬ 
mented  a  random  ordering  algorithm,  that  chooses  parity  check  orderings  with 
equal  probability  for  each,  with  the  (396,5,6)  and  (396,3,6)  codes.  Finally,  to 
investigate  how  typical  c39656  was  compared  to  other  (396,5,6)  codes,  objec¬ 
tive  functions  1.1  and  3  were  used  with  b39656,  another  (396,5,6)  low-density 
code.  Graphs  of  the  results  are  shown  in  Figures  3.4  through  3.10.  Note  that 
all  codes  were  chosen  to  have  the  property  that  no  two  parity  check  sets  con¬ 
tain  more  than  one  digit  in  common,  as  discussed  in  Section  1.4.  A  computer 
program  to  generate  such  codes  is  included  in  Section  A.l. 

To  illustrate  how  the  graphs  were  obtained.  Figure  3.3  contains  a  table  of 
simulation  results  for  objective  function  1.1  used  with  code  c39656.  Each 
data  point  was  generated  using  a  procedure  similar  to  that  used  in  Section  2.2; 
the  differences  are  described  below.  The  quantities  c,  s,  and  s  are  the  same 
as  before,  and  the  same  stopping  criterion  was  used.  However,  the  sequential 
decoder  used  here  can  declare  a  decoding  failure,  unlike  the  stack  algorithm 
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Figure  3.3.  Simulation  results  for  objective  function  1.1  used  with  code 
c39656.  Additional  trials  were  performed  to  obtain  better  estimates  of  pDF 
for  e  =  15  and  e  =  20.  For  e  =  15,  there  were  zero  decoding  failures  out  of 
4000  trials,  and  for  e  =  20,  there  were  3  decoding  failures  out  of  4000  trials. 
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Figure  3.4.  Simulation  results  for  objective  functions  1.1,  1.2,  1.3,  and  1.4 
used  with  a  (396,5,6)  low-density  code.  For  both  objective  functions  1.1  and 
1.2,  with  25  channel  errors,  there  were  zero  decoding  failures  out  of  4(XX)  tri¬ 
als. 
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code  C39656 


code  C39656 


Figure  3.5.  Simulation  results  tor  objective  functions  1.2,  2,  3,  and  4  used 
with  a  (396.5,6)  low-density  code. 
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Figure  3.7.  Simulation  results  for  random  ordering  and  objective  functions 
1.2,  7.1,  and  7.2  used  with  a  (396,5,6)  low-density  code. 


probability  of  decoding  failure  average  computation 


code  C39636 


* -  1.2 

■o—  random 


code  C39636 


•a -  random 


Figure  3.8.  Simulation  results  for  random  ordering  and  objective  functions 
1.2  and  3  used  with  a  (396,3,6)  low-density  code. 
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code  C40048 


o -  1.2 


code  c40048 


Figure  3.9.  Simulation  results  tor  objective  tunctions  1.2  and  3  used  with  a 
(400,4,8)  low-density  code.  For  objective  function  1.2  with  10  channel  errors, 
there  were  zero  decoding  failures  out  of  2477  trials. 
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Figure  3.10.  Simulation  results  tor  objective  functions  1.2  and  3  used  with 
two  (396,5,6)  low-density  codes,  b39656  and  c39656. 
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used  on  the  BEC.  As  a  result,  the  computer  recorded  NF,  the  number  of 
declared  decoding  failures,  as  well  as  N£,  the  number  of  decoding  errors,  and 
Na,  the  number  of  aborted  trials.  In  this  case,  a  trial  was  aborted  if  the  com¬ 
putation  reached  106;  this  higher  value  was  used  because  steps  performed  by 
the  sequential  decoder  used  here  are  simpler  than  those  of  the  stack  algo¬ 
rithm.  The  estimated  probability  of  decoding  failure  is  now  given  by 
pDF  =  where  T  is  the  total  number  of  trials. 

The  functions  that  minimized  the  average  computation  were  5.1  and  6.1, 
and  the  results  for  these  two  were  nearly  equal.  Apart  from  the  random  ord¬ 
ering  algorithm,  7.1  and  7.2  performed  worst.  The  average  computation  for 
7.2  was  as  much  as  12  times  as  great  as  that  of  6.1.  However,  all  the  objec¬ 
tive  functions  performed  better  than  random  ordering,  which  required  as 
much  as  16  times  the  computation  required  by  6.1.  With  respect  to  pDF,  the 
probability  of  decoding  failure,  objective  function  1.2  performed  best,  closely 
followed  by  1.1.  Random  ordering  was  worst,  followed  by  7.1  and  7.2. 

Recall  that  pDF  was  defined  to  include  decoding  failures,  decoding  errors, 
and  trials  aborted  due  to  too  much  computation.  As  with  the  BEC  results, 
there  was  a  surprising  lack  of  decoding  errors.  A  decoding  error  occurs  when 
the  decoder  completes  execution  and  produces  an  output,  but  the  output  is 
incorrect.  Out  of  a  total  of  112,898  trials,  only  two  resulted  in  decoding 
errors.  Both  errors  occurred  using  objective  function  1.2  with  the  (396,3,6) 
code  and  25  channel  errors.  This  was  the  weakest  code  used;  also,  pDF  was 
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0.634,  hence  this  algorithm  would  probably  not  be  used  with  this  much  chan¬ 
nel  noise  anyway.  Therefore,  if  these  decoding  algorithms  are  used  on  a 
channel  with  feedback  and  retransmission  capabilities,  one  can  achieve  decod¬ 
ing  error  rates  several  orders  of  magnitude  smaller  than  pDF-  However,  the 
effective  information  rate  would  be  lower  than  the  code  rate,  due  to 
retransmissions. 

The  relative  performance  of  random  ordering  and  functions  1.2  and  3 
observed  with  the  (396,3,6)  and  (400,4,8)  codes  was  the  same  as  that  observed 
with  code  c39656.  This  reinforces  the  hypothesis  that  an  ordering  algorithm 
that  performs  well  for  one  value  of  j  and  k  will  also  perform  well  for  other 
values.  The  average  computation  for  the  (400,4,8)  code  was  roughly  eight 
times  greater  than  that  of  the  (396,3,6)  code,  while  pDF  was  greater  for  high 
channel  noise  and  less  for  low  channel  noise.  Note  that  both  these  codes 
have  the  same  designed  code  rate,  1/2.  Thus,  it  seems  that  increasing  k  will 
dramatically  increase  the  average  computation,  even  when  the  code  rate  is 
kept  fixed. 

With  regard  to  b39656,  the  second  (396,5,6)  code,  the  results  agreed  very 
well  with  those  obtained  with  c39656.  This  reinforces  the  claim  that  c39656  is 
a  typical  (396,5,6)  low-density  code.  It  should  be  stressed  that  b39656  and 
c39656  were  chosen  randomly,  without  regard  for  their  performance. 

From  the  simulation  results,  it  appears  that  objective  function  1.2  per¬ 
formed  best  overall.  Its  average  computation  was  comparable  to  that  of  the 
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minimum  observed  values,  and  its  probability  of  decoding  failure  was  lowest. 
This  leads  one  to  consider  what  characterizes  a  good  objective  function.  It 
seems  unlikely  that  V{  itself  is  an  exceptionally  good  quantitative  measure  of 
digit  reliability,  even  though  it  was  used  this  way  in  function  1.2.  Why  should 
a  digit  involved  in  three  violated  parity  checks  be  1.5  times  as  unreliable  as  a 
digit  involved  in  two?  Instead,  it  seems  that  function  1.2  performed  well 
because  its  measure  of  digit  reliability  was  discrete.  This  allowed  the  ordering 
algorithm  to  consider  a  group  of  parity  checks  with  relatively  good  digit  relia¬ 
bilities,  and  of  these,  choose  the  one  with  minimum  \N\.  With  this 
approach,  the  digit  reliability  measure  need  not  be  extremely  accurate,  and  the 
exact  tradeoff  between  maximizing  reliability  and  minimizing  |  N  |  need  not  be 
specified.  Thus,  one  would  expect  that  discretizing  any  of  the  reliability  meas¬ 
ures  used  by  the  relatively  good  objective  functions  -  namely,  3,  4,  5,  and  6  - 
together  with  breaking  ties  by  minimizing  \N\,  will  result  in  a  good  ordering 
algorithm. 

What  conclusions  can  we  draw  concerning  the  feasibility  of  this  approach 
at  rates  greater  than  RQ,  the  computational  cutoff  rate  for  sequential  decod¬ 
ing?  Note  that  this  discussion  applies  only  to  the  binary  symmetric  channel. 
First,  we  consider  low-density  codes  with  =  5  and  k  —  6,  which  have 
designed  code  rate  R=  l—j/k  =  1/6.  It  is  well  known  that  the  formula  for 
R0,  given  in  Section  1.1,  applied  to  the  BSC  yields 
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R0  =  l-  log2  [l-f2Vp(l_p)]  ,  (3.16) 

where  p  is  the  channel  error  probability  ana  RQ  is  measured  in  bits  per  chan¬ 
nel  use.  The  value  of  p  corresponding  to  RQ  =  1/6  is  p  =  0.1882.  For  block- 
length  n  —  396,  this  leads  to  an  expected  number  of  errors  per  codeword  given 
by  «p=74.5.  To  decide  how  many  errors  per  codeword  can  be  handled  by 
the  best  BSC  algorithm,  we  used  a  different  criterion  than  for  the  BEC.  This 
was  done  because  average  computation  on  the  BSC  did  not  increase  as 
abruptly  as  on  the  BEC.  The  criterion  used  here  is  that  pDF  should  be  less 
than  0.1.  Then,  at  most  30  errors  per  codeword  can  be  handled  by  the  best 
algorithm  presented  here.  This  is  summarized  in  Table  3.1.  Thus,  it  does  not 
appear  feasible  to  use  these  algorithms  at  rates  greater  than  RQ  for  rate  1/6 
codes. 

However,  low-density  codes  with  different  values  of  j  and  k  are  more 
promising.  Table  3.1  contains  data  for  the  (396,3,6)  and  (400,4,8)  codes  used 
here.  Both  have  designed  rate  /?  =  1/2.  As  above,  PDF<  0.1  was  used  to 
determine  feasibility.  The  results  for  these  codes  are  very  close  to  the  max¬ 
imum  performance  of  sequential  decoding.  This  is  quite  good,  because  the 
low-density  codes  were  chosen  at  random,  without  regard  for  their  perform¬ 
ance,  and  the  result  for  sequential  decoding  is  a  maximum  for  all  tree  codes. 
More  study,  including  simulations  using  codes  with  longer  blocklengths,  would 
show  more  accurately  whether  these  algorithms  allow  one  to  exceed  RQ. 
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Table  3.1.  Comparison  of  SDR  algorithms  and  sequential  decoding  for  the 
binary  symmetric  channel. 


Code  parameters 

n  j  k 

Rate 

R 

Achievable  np 
with  SDR 
(approximate) 

Max.  crossover 
prob.  for  sequential 
decoding 

Max.  np  for 
sequential 
decoding 

39  6 

5 

6 

30 

0.1882 

74.5 

396 

3 

6 

17 

0.0449 

17.8 

400 

4 

8 

17 

0.0449 

18.0 
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Since  these  algorithms  perform  better  at  rate  1/2  than  rate  1/6,  this  leads 
one  to  question  whether  they  perform  even  better  at  higher  rates.  More  work 
is  required  to  answer  this  question.  It  should  be  stressed  that  the  results 
given  here  are  only  lower  bounds  for  what  is  achievable  with  SDR  algorithms. 
Other  ordering  algorithms,  with  possibly  completely  different  structures,  may 
perform  better. 

Some  features  that  limit  this  approach,  that  are  not  encountered  in  tradi¬ 
tional  sequential  decoding,  include  the  following.  As  pointed  out  in  Section 
1.2,  the  codeword  tree  for  a  low-density  code  grows  more  rapidly  near  the 
beginning  than  near  the  end.  This  is  a  drawback  for  the  same  reasons  dis¬ 
cussed  in  Section  2.2.  In  addition,  low-density  codes  may  not  have  as  good 
distance  structure  as  arbitrary  tree  codes.  Another  important  point  is  that  the 
BSC  is,  in  a  sense,  the  worst  possible  channel  for  SDR  algorithms.  Each 
digit  at  the  channel  output,  taken  by  itself,  is  equally  unreliable  -  unlike,  for 
example,  the  binary  erasure  channel  and  the  additive  white  Gaussian  noise 
channel.  This  makes  it  difficult  to  generate  a  good  codeword  tree  ordering. 

3.3  Backtracking  Procedures  and  a  New  Sequential  Decoding  Algorithm 

Without  modification,  the  Fano  sequential  decoding  algorithm  (p.  269  of 
[5])  does  not  work  well  with  low-density  codes.  The  problem  occurs  when  the 
decoder  decides  it  is  on  the  wrong  path  due  to  a  wrong  move  earlier  in  the 
codeword  tree.  This  happens  when  none  of  the  branches  at  the  current  level 
of  the  tree  are  close  to  the  corresponding  received  digits;  each  branch  causes 
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the  path  metric  to  fall  below  the  threshold.  In  the  Fano  algorithm,  the 
decoder  moves  back  one  level  and  tries  to  move  laterally.  If  it  cannot  find  an 
acceptable  path,  it  moves  back  another  level  and  again  tries  to  move  laterally. 
This  continues  until  the  decoder  reaches  the  most  recent  level  at  which  the 
threshold  was  raised,  at  which  point  it  lowers  the  threshold  and  tries  to  move 
forward  again. 

This  procedure  works  well  for  convolutional  codes,  because  if  the 
decoder  makes  a  wrong  move,  the  following  levels  of  the  codeword  tree 
quickly  become  independent  of  the  received  data.  Thus,  when  the  decoder 
makes  a  wrong  move,  it  will  realize  this  within  a  few  levels,  and  backtracking 
one  level  at  a  time  will  quickly  find  the  problem.  This  situation  does  not  hold 
for  low-density  codes.  In  a  low-density  code’s  codeword  tree,  it  is  possible  to 
have  the  two  subtrees  following  a  correct  move  and  an  incorrect  move  be 
identical  for  many  levels  (for  example,  50  or  more  levels  in  a  400  blocklength 
code). 

This  situation  occurs  because  the  labels  of  the  branches  at  a  given  level 
in  the  codeword  tree  depend  only  on  the  values  of  the  few  digits  in  the  level’s 
o-set.  Recall  that  each  level  in  a  low-density  code’s  codeword  tree 
corresponds  to  a  parity  check.  The  new  digits  at  a  level  are  those  digits  that 
are  involved  in  the  corresponding  parity  check,  but  not  in  any  of  the  previous 
parity  checks.  The  old  digits  at  a  level  are  the  remaining  digits  involved  in  the 
level’s  parity  check;  they  have  already  been  assigned  values  earlier  in  the  tree. 
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As  in  Section  1.2,  the  set  of  new  digits  at  a  level  is  called  the  level’s  n-set, 
and  the  set  of  old  digits  is  called  the  o-set.  The  parity  of  a  level’s  old  digits 
determines  what  values  can  be  assigned  to  a  level’s  new  digits. 

When  a  wrong  move  occurs,  the  two  subtrees  originating  from  the  wrong 
move  and  the  correct  move  will  be  identical  until  one  of  the  digits  assigned 
incorrectly  in  the  wrong  move  occurs  in  some  level’s  o-set.  It  is  true  that  the 
ordering  algorithm  favors  parity  checks  with  large  o-sets,  so  that  once  a  parity 
check  containing  a  given  digit  is  chosen,  other  parity  checks  containing  that 
digit  are  more  likely  to  be  chosen.  However,  given  an  occurrence  of  a  digit  at 
a  level  in  the  codeword  tree,  there  is  no  guarantee  for  when  its  next 
occurrence  will  be. 

Instead  of  backtracking  one  level  at  a  time,  the  decoding  algorithm  could 
jump  back  to  one  of  the  levels  that  directly  influence  the  current  level.  These 
levels,  called  significant  parents,  are  the  places  where  the  digits  in  the  o-set  of 
the  current  level  were  initially  assigned.  Since  we  consider  only  low-density 
codes  for  which  no  two  parity  check  sets  contain  more  than  one  digit  in  com¬ 
mon,  the  number  of  preceding  levels  that  directly  influence  the  current  level 
will  equal  the  number  of  digits  in  the  current  level’s  o-set.  However,  since 
this  means  there  will  be  typically  more  than  one  significant  parent,  it  is  not 
immediately  apparent  how  to  incorporate  this  into  the  Fano  algorithm. 

A  sequential  decoding  algorithm  that  utilizes  significant  parents  is 
presented  in  Section  A. 3.  A  description  of  the  algorithm  is  given  below, 
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followed  by  a  discussion  of  its  features  and  a  bound  on  its  computational 
complexity. 

3.3.1  The  sequential  decoding  algorithm 

This  algorithm  is  similar  to  the  Fano  algorithm  and  unlike  the  stack  algo¬ 
rithm  (Section  7.2.7  of  Clark  and  Cain  [3])  in  that  it  stores  only  a  single  path 
through  the  codeword  tree.  As  in  the  Fano  algorithm,  the  decoder  calculates 
a  metric  value  for  each  branch  of  the  path  it  takes.  The  metric  is  given 
below. 


If  n.  >  0, 


r,(x,,y()  -  £ 

m- 1 


1q  -  _ 


(3.17) 


If  =  0, 


r,= 


0 


—  co 


if  the  parity  check  at  level  i  is  satisfied, 

(3.18) 

otherwise. 


where 

n 

F( (x, ,y • )  =  the  metric  value  of  the  branch  at  level  i  with  label  =  (.v;  m)m=1 , 

n. 

when  the  corresponding  received  digits  are  y-  =  (y-  OT)m'_1 
ni  =  the  number  of  new  digits  at  level  i  in  the  codeword  tree 
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V,  m  =  the  error  statistic  associated  with  y. 

=  the  number  of  violated  parity  checks  involving  yi  m 
r.  =  the  local  rate  of  the  codeword  tree  at  level  i,  defined  below. 

The  probabilities  are  calculated  by  assuming  that  the  xi  m  are  independent  and 
P(.vf  m  =  0)  =  P^  m  =  1)  =  1/2.  The  relevant  formulas  are  given  in  Section 
3.1.  This  metric  is  the  same  as  the  Fano  metric  (given  by  Ln  in  Section  VII 
of  Fano  [4])  except  for  the  inclusion  of  Vi  m  and  r.,  and  the  rule  for  determin¬ 
ing  T.  when  ni  —  0. 

The  error  statistic  V-  m  is  discussed  in  Section  3.1.  It  is  included  as  a 

i  ,m 

form  of  side  information  to  estimate  the  reliability  of  a  received  digit.  In  an 
( n,j,k )  low-density  code,  there  are  only  j+ 1  values  that  Vi  m  can  assume. 
For  each  of  these  values,  the  log  term  above  can  have  two  values,  depending 
on  whether  or  not  y •  m  equals  x- m.  Thus,  the  log  term  will  assume  only 
2(;'  +  l)  different  values.  They  can  be  efficiently  stored  in  a  table  and  need 
not  be  recalculated. 

The  local  rate,  r(,  replaces  the  fixed  bias  term  in  the  Fano  metric.  The 
bias  term  usually  equals  R,  the  code  rate.  The  local  rate  is  used  because  the 
codeword  tree  for  a  low-density  code  does  not  grow  with  constant  rate,  as  dis¬ 
cussed  in  Section  1.2.  The  local  rate  is  given  by 


(3.19) 
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where  ni  is  defined  as  above,  and  /(  is  the  number  of  consecutive  levels  s 
preceding  i  with  ns  =  0.  In  other  words,  i  —  —  1  is  the  first  level  preceding  i 

with  ni  >  0. 

This  definition  is  used  because  of  the  heuristic  justification  for  the  Fano 
metric  given  in  Section  VII  of  [4].  It  seems  that  this  justification  implicitly 
uses  the  fact 

d, 

£*,  =logS,  ,  (3.20) 

t~  l 

where  di  is  the  number  of  digits  encountered  in  the  tree  up  to  and  including 
level  i,  Bt  is  the  bias  term  for  digit  t,  and  S-  is  the  size  of  the  tree  at  level  i. 

djR 

For  a  fixed  rate  tree  code,  Si  —  2  ,  thus  Bt  —  R  satisfies  (3.20).  For  a  low- 

density  code’s  tree,  Equations  (1.2)  and  (1.3)  give 

i 

log25i  =  £  (ni+  ai  -  !)  >  (3.21) 

w— 1 

where  a-=  1  if  parity  check  i  is  redundant,  given  the  preceding  parity  checks, 
and  ct-  =  0  otherwise.  If  one  ignores  oti  and  divides  the  bias  term  contribution 
of  a  level  equally  among  that  level’s  digits,  one  obtains  (3.19).  For  a  ran¬ 
domly  chosen  code,  redundant  parity  checks  will  be  rare,  thus  ignoring  a(. 
should  not  have  a  significant  effect.  It  was  found  in  simulations  that  using  ri 
as  the  bias  term  worked  much  better  than  using  R. 
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The  decoder  can  be  in  one  of  three  states  -  Forward  mode,  Backtrack 
mode,  or  Reset  mode.  Initially,  the  decoder  is  in  Forward  mode.  Starting 
from  the  root  node  of  the  codeword  tree,  the  decoder  moves  forward  one 
level  at  a  time.  At  each  level,  the  decoder  chooses  the  first  outgoing  branch 
with  nonnegative  branch  metric.  Note  that  it  considers  the  branch  metric,  not 
the  cumulative  metric.  However,  if  the  decoder  reaches  a  level  where  all  out¬ 
going  branches  have  negative  metric  values,  it  enters  Backtrack  mode. 

The  level  where  the  decoder  enters  Backtrack  mode  is  denoted  by 
bt_level.  The  purpose  of  entering  this  mode  is  to  find  a  new  path  in  the  tree 
that  has  nonnegative  branch  metric  at  btjevel.  However,  as  discussed  above, 
this  cannot  be  done  efficiently  by  backtracking  one  level  at  a  time.  Instead, 
the  decoder  makes  a  list  of  the  significant  parents  of  bt_level.  It  assumes  it 
made  a  wrong  move  at  one  of  the  parent  levels,  and  tries  to  correct  this  by 
making  changes  at  each  of  these  levels,  one  at  a  time.  It  starts  by  jumping 
back  to  the  first  parent  on  its  list  and  choosing  the  first  outgoing  branch  - 
unless  the  original  path  followed  the  first  branch,  in  which  case  it  chooses  the 
second.  Note  that  it  makes  this  choice  even  if  the  resulting  branch  metric  is 
negative. 

The  decoder  then  works  its  way  back  up  to  bt_level  in  the  same  way  as  in 
Forward  mode,  with  the  following  exception.  If  it  encounters  a  level  where  all 
outgoing  branches  have  negative  metric,  it  chooses  the  branch  with  greatest 
metric  and  continues  forward,  instead  of  starting  a  new  backtrack  procedure. 
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When  the  decoder  reaches  bt_level,  it  stores  the  value  of  the  cumulative  path 
metric. 

This  process  is  repeated  for  the  other  outgoing  branches  at  the  first 
parent  level,  and  for  all  the  parent  levels,  except  that  any  branch  choice  which 
was  on  the  original  path  is  skipped.  Also,  the  last  branch  setting  tried  for 
one  parent  must  be  reset  before  changing  the  next  parent.  The  decoder  then 
decides  whether  to  accept  any  of  these  changes.  It  considers  the  change  with 
greatest  cumulative  metric  at  bt_level.  If  this  metric  value  is  greater  than  the 
original,  it  accepts  the  change;  otherwise,  no  change  is  accepted. 

At  this  point,  the  decoder  enters  Reset  mode.  The  last  change  is  reset  to 
its  original  value,  and  any  accepted  change  is  implemented.  In  addition,  if  a 
change  is  accepted,  its  value  is  recorded.  This  insures  that  the  change  is 
remembered  if  the  decoder  passes  through  the  changed  level  again,  due  to 
another  backtrack.  Note  this  does  not  protect  the  level  from  further  change 
if  it  occurs  as  the  significant  parent  of  a  new  bt_level. 

If  there  is  now  an  outgoing  branch  at  bt_level  with  nonnegative  metric, 
the  decoder  chooses  the  first  such  branch  and  reenters  Forward  mode.  If  all 
branch  metrics  are  still  negative,  two  actions  are  possible,  depending  on  the 
value  of  ni  for  i  =  bt_level.  If  ni  =  0,  then  the  parity  check  at  bt Jevel  is  still 
unsatisfied;  the  decoder  declares  a  decoding  failure  and  halts.  If  ni  >  0,  the 
decoder  chooses  the  outgoing  branch  with  greatest  metric  and  reenters  For¬ 
ward  mode.  This  last  event  does  not  necessarily  mean  the  decoder  is  on  the 
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wrong  path;  channel  noise  can  and  most  probably  will  result  in  negative 
branch  metrics  even  on  the  correct  path. 

The  decoder  continues  in  this  fashion  until  it  tries  to  move  forward  from 
the  last  level  in  the  tree,  at  which  point  it  is  done.  Its  output  is  the  valid 
codeword  corresponding  to  its  final  path. 

Various  shortcuts  were  used  in  the  implementation  of  this  algorithm.  For 
example,  recall  that  one  digit  in  a  parent’s  n-set  corresponds  to  a  digit  in  the 
child’s  o-set.  To  limit  computation,  the  decoder  tried  a  branch  setting  at  a 
parent  only  if  the  setting  changed  the  value  of  the  digit  in  its  child’s  o-set. 
Otherwise,  the  branch  metric  at  the  child  would  not  be  raised,  unless  there 
was  some  other  chain  of  parent-child  relations  leading  from  the  parent  to  the 
child,  an  unlikely  occurrence.  For  another  shortcut,  the  decoder  used  only 
one  pass  to  reset  the  last  change  at  one  parent  and  set  the  first  change  at  the 
next. 

3.3.2  Discussion 

The  sequential  decoding  algorithm  described  above  is  a  fairly  straightfor¬ 
ward  way  to  incorporate  the  idea  of  jumping  back  directly  to  a  level’s  signifi¬ 
cant  parents.  However,  several  other  possibilities  were  considered,  and  it 
seems  appropriate  to  comment  on  some  features  of  the  present  algorithm. 

In  what  follows,  a  backtrack  procedure  is  a  decision  by  the  decoder  that 
it  should  change  a  branch  setting  at  one  of  the  significant  parents  of  a  level. 


In  the  sequential  decoding  algorithm  described  above,  there  is  never  more 
than  one  active  backtrack  procedure  at  any  given  time.  When  the  decoder 
enters  Backtrack  mode,  it  either  accepts  a  change  in  its  path  or  it  decides  that 
no  change  is  necessary.  In  either  case,  the  issue  is  resolved  and  the  decoder 
will  never  again  enter  Backtrack  mode  using  the  same  bt_level.  The  algorithm 
is  designed  this  way  because  the  situation  would  become  quite  complicated  if 
multiple  unresolved  backtrack  procedures  were  allowed.  To  see  why,  suppose 
that  several  unresolved  backtrack  procedures  are  active,  and  the  decoder 
decides  that  it  is  on  the  wrong  path.  The  decoder  would  have  to  decide  which 
set  of  parents  t^  continue  changing  or  whether  to  start  a  new  backtrack  pro¬ 
cedure.  In  addition,  the  different  backtrack  procedures  could  perturb  each 
other.  For  example,  a  change  made  by  one  procedure  could  reassign  a  digit 
in  the  o-set  of  a  parent  in  another  procedure.  This  changes  the  set  of  possi¬ 
ble  branch  settings  available  at  the  parent,  which  could  cause  a  good  change 
in  the  second  procedure  to  look  bad  or  cause  the  second  procedure  to  skip 
over  a  good  change. 

Another  feature  of  the  decoding  algorithm  is  that,  unlike  the  Fano  algo¬ 
rithm,  the  decision  to  start  backtracking  is  based  on  branch  metrics,  not  the 
cumulative  path  metric.  As  discussed  before,  the  two  subtrees  leading  from  a 
correct  move  and  an  incorrect  move  may  be  the  same  for  many  levels.  If  the 
decoder  makes  a  wrong  move,  but  moves  through  the  incorrect  subtree  along 
the  path  that  matches  the  correct  one,  the  path  metric  may  accumulate  a  large 
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positive  increase.  This  could  drown  out  negative  branch  metrics  until  it  is  too 
late  to  discover  the  initial  wrong  move. 

Recall  that  in  a  backtrack  procedure,  all  potential  changes  are  tried 
before  a  final  decision  is  made.  This  may  seem  wasteful;  one  could  stop 
backtracking  immediately  after  an  acceptable  choice  is  discovered.  However, 
suppose  that  a  level’s  position  in  the  tree  is  very  close  to  one  of  its  parents. 
Then  it  is  quite  probable  that  a  change  at  this  parent  would  affect  the  metric 
value  only  at  the  parent  level  and  the  child  level.  If  the  branch  metric  at  the 
parent  level  decreases,  while  the  metric  at  the  child  level  increases,  the  net 
effect  could  easily  be  positive  even  if  the  change  is  incorrect.  Testing  all  pos¬ 
sible  changes  will  insure  that  the  correct  change  is  considered,  if  it  exists. 
Ruling  out  changes  in  “close”  parents  would  not  work  well  either,  because  it 
was  found  that  too  many  necessary  changes  were  missed. 

Another  possible  remedy  for  ba<  =  changes  caused  by  close  parents  is  to 
evaluate  changes  at  a  level  beyond  btjevei,  denoted  by  stopjevel.  However, 
it  is  difficult  to  determine  where  stopjevel  should  be  located.  One  rule  that 
was  implemented  was  to  make  a  list  of  levels  with  negative  branch  metrics 
beyond  btjevei  and  choose  stopjevel  to  be  the  entry  at  a  fixed  position  in 
the  list.  This  tends  to  insure  that  there  will  be  some  coupling  between  the 
changes  made  to  a  close  parent  and  the  levels  traversed  by  the  decoder. 
However,  it  was  found  that  wrong  moves  after  btjevei  introduced  too  much 
uncertainty  into  the  path  metric  value  at  stopjevel. 


92 


Finally,  two  other  significant  changes  in  the  algorithm  were  considered. 
Recall  that  the  branch  metric  value  is  — oo  for  an  unsatisfied  parity  check  at  a 
level  with  ni  =  0.  Instead,  this  could  be  a  finite  negative  number,  and  one 
need  not  immediately  declare  a  decoding  failure  if  a  backtrack  procedure  fails 
to  remedy  the  situation.  This  would  give  the  sequential  decoder  more 
chances  to  find  any  incorrect  moves.  However,  this  feature  rarely  decoded  a 
vector  that  was  not  decodable  by  the  original  algorithm,  and  it  usually 
required  significantly  more  computation.  Another  possible  change  in  the 
algorithm  is  to  backtrack  through  more  than  one  level  of  parents;  that  is,  con¬ 
sider  the  parent  of  a  parent,  and  so  on.  However,  this  would  cause  a  huge 
increase  in  the  number  of  changes  to  consider. 

3.3.3  A  computation  bound 

The  sequential  decoding  algorithm  described  above  has  the  nice  property 
that  the  decoder  always  moves  forward  through  the  codeword  tree,  in  a  cer¬ 
tain  sense.  More  specifically,  the  decoder  can  enter  Backtrack  mode  from 
any  given  level  at  most  once.  From  this  property,  one  can  easily  obtain  a 
bound  on  the  number  of  forward  moves  performed  by  the  decoder.  Here,  a 
forward  move  means  any  time  the  decoder  moves  from  one  level' to  the  next, 
regardless  of  whether  the  decoder  is  in  Forward,  Backtrack,  or  Reset  mode. 
The  bound  is  given  below. 
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F  <  n1(l—R)2k2k~1  ,  (3.22) 

where 

F  =  the  total  number  of  forward  moves 
performed  by  the  sequential  decoder 
(n,  j ,k)  =  low-density  code  parameters 

R  =  the  designed  information  rate  of  the  code 
-  l-j/k. 


Proof:  Suppose  the  decoder  enters  Backtrack  mode  with  bt_level  =  i, 
where  i  is  a  given  level  in  the  codeword  tree.  Let  /.  be  the  number  of  for¬ 
ward  moves  performed  by  the  decoder  before  it  reenters  Forward  mode. 
Then 


/,  <iai2k-'+i  , 


(3.23) 


where  ai  is  the  number  of  old  digits  at  level  i,  that  is,  the  number  of  digits 

assigned  earlier  in  the  tree.  The  first  term  is  present  because:  (1)  ai  is  the 

k—\ 

number  of  parents  of  level  i,  (2)  there  are  at  most  2  possible  branch  set¬ 
tings  for  each  parent,  and  (3)  the  number  of  levels  between  a  parent  level  and 
bt_level  is  strictly  bounded  by  i.  The  second  term  bounds  the  number  of  for¬ 
ward  moves  performed  in  Reset  mode. 


A  given  level  can  be  bt_level  at  most  once,  thus  fbr>  the  total  number  of 


forward  moves  performed  during  Backtrack  or  Reset  modes,  satisfies 
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nj/k 

FBr  <  £  f  , 

(-1 

nj/k 

<  J2(ia,2k-'+i) 

i- 1 

<  \nj(nj/k+l)  2fc_1  +  j(nj/k)(nj/k+l)  ,  (3.24) 

since  nj/k  is  the  number  of  levels  in  the  codeword  tree,  and  ai  <  k  for  all  i. 
The  number  of  forward  moves  performed  during  Forward  mode,  Ff>  equals 
the  number  of  levels  in  the  codeword  tree,  or  nj/k.  Using  F  =  Fbr+Ff  and 
R  =  1 —j/k,  we  obtain  the  desired  result.  □ 

This  bound  is  quite  loose  because  it  assumes  that  the  decoder  enters 
Backtrack  mode  from  every  level  in  the  codeword  tree.  However,  the  bound 
is  much  smaller  than  the  total  number  of  nodes  in  the  tree,  which  is  on  the 
order  of  n2nR . 

In  practice,  this  decoding  scheme  would  probably  be  used  only  if  the 
noise  level  was  low  enough  so  that  relatively  few  backtrack  procedures  would 
be  necessary.  The  maximum  noise  level  would  depend  on  the  code  rate  and 
the  maximum  tolerable  amount  of  computation. 
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CHAPTER  4 
CONCLUSIONS 

4.1  Summary  of  Results 

[ 

I 

In  this  thesis,  we  presented  several  decoding  algorithms,  called  sequential 

' 

decoding  with  reordering  (SDR)  algorithms.  These  algorithms  are  modifica¬ 
tions  of  standard  sequential  decoding;  they  have  the  general  structure  shown 
in  Figure  1.2.  These  modifications  were  done  in  an  attempt  to  operate  at 
rates  greater  than  R0,  the  computational  cutoff  rate  for  sequential  decoding. 

The  SDR  algorithms  have  two  parts  -  an  ordering  algorithm  and  a 
sequential  decoder.  The  ordering  algorithm  chooses  an  ordering  of  the  parity 
checks  that  define  the  code  being  used.  The  sequential  decoder  uses  this  ord¬ 
ering  to  generate  a  codeword  tree  and  searches  this  tree  to  obtain  the  decoder 
output.  The  SDR  algorithms  are  used  with  low-density  codes,  a  class  of 
block  codes.  In  this  respect  they  differ  from  standard  sequential  decoding, 
which  can  be  used  with  any  tree  code.  Low-density  codes  are  used  because 
one  can  reorder  their  associated  codeword  trees,  and  the  resulting  trees  have 
bounded  growth  rate. 

We  presented  SDR  algorithms  for  two  channels  -  the  binary  erasure 
channel  (BEC)  and  the  binary  symmetric  channel  (BSC).  For  the  BEC,  an 
upper  bound  was  derived  for  the  number  of  nodes  visited  by  the  sequential 
decoder,  as  a  function  of  the  parity  check  ordering  and  the  channel  erasure 
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pattern.  This  bound  motivated  the  construction  of  an  ordering  algorithm, 
called  the  minimum  new  erasure  (MNE)  algorithm. 

The  MNE  algorithm  was  used  in  simulations  with  the  stack  algorithm,  a 
standard  sequential  decoding  algorithm.  The  MNE  algorithm  performed 
much  better  than  random  ordering.  However,  when  used  with  a  (396,5,6) 
low-density  code,  with  designed  rate  R  =  1/6,  and  a  (396,4,6)  low-density 
code,  with  /?=  1/3,  the  MNE  algorithm  did  not  exceed  the  R0  bound,  which 
represents  the  best  that  standard  sequential  decoding  can  achieve.  The  per¬ 
formance  was  better  for  a  (396,3,6)  code,  with  R=  1/2.  In  this  case,  the  algo¬ 
rithm  roughly  matched  the  RQ  bound.  Simulations  with  longer  blocklengths 
would  indicate  more  clearly  whether  the  RQ  bound  is  exceeded. 

Also  on  ihe  BEC,  simulations  verified  the  hypothesis  that  for  a  fixed 
designed  code  rate,  the  erasure  correcting  ability  of  the  MNE  algorithm 
increases  linearly  with  blocklength.  This  behavior  is  characteristic  of  standard 
sequential  decoding,  but  is  not  common  among  existing  practical  decoding 
schemes  for  block  codes.  In  addition,  in  the  feasible  erasure  region  for  the 
MNE  algorithm,  the  average  computation  of  the  sequential  decoder  portion 
increased  linearly  with  blocklength. 

For  the  BSC,  several  ordering  algorithms  were  presented,  as  well  as  a 
new  sequential  decoder.  All  the  ordering  algorithms  have  structures  similar  to 
that  of  the  MNE  algorithm;  they  differ  in  their  choice  of  objective  function. 
A  list  of  various  objective  functions  is  shown  in  Figure  3.2.  Through 
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simulation,  it  was  found  that  function  1.2  performed  best.  It  performed  signi¬ 
ficantly  better  than  random  ordering.  Using  a  (396,5,6),  R=  1/6  low-density 
code,  the  best  BSC  algorithm  did  not  exceed  the  RQ  bound.  However,  as 
with  the  BEC,  performance  improved  at  higher  code  rates.  For  both  a 
(396,3,6)  and  a  (400,4,8),  each  with  R=  1/2,  the  R0  bound  was  roughly 
matched.  Again,  simulations  with  longer  blocklengths  would  provide  more 
information. 

For  both  the  BEC  and  BSC  algorithms,  decoding  errors  were  much  less 
frequent  than  decoding  failures.  This  feature  could  be  used  to  advantage  on  a 
channel  with  feedback  and  retransmission  capabilities. 

The  BSC  algorithms  were  used  with  a  new  sequential  decoding  algorithm, 
described  in  Section  3.3.  In  a  low-density  code’s  codeword  tree,  the  two  sub¬ 
trees  originating  from  a  correct  move  and  an  incorrect  move  may  appear  the 
same  for  many  levels,  apart  from  the  different  assignments  made  at  the  level 
where  they  originally  diverge.  As  a  result,  backtracking  one  level  at  a  time,  as 
in  the  Fano  algorithm,  can  be  very  inefficient.  For  a  low-density  code,  the 
labels  at  a  given  level  in  the  codeword  tree  depend  on  the  assignments  at  a 
small  group  of  preceding  levels,  called  the  “significant  parents.”  The  new 
sequential  decoding  algorithm  is  able  to  backtrack  directly  to  a  level’s  signifi¬ 
cant  parents.  As  a  result,  the  algorithm  may  be  applicable  to  other  tree 
search  problems  where,  instead  of  a  single  level,  it  is  desirable  to  consider 
several  levels  as  possible  backtrack  destinations. 
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The  sequential  decoder  uses  a  modified  version  of  the  Fano  metric.  It 
differs  from  the  standard  Fano  metric  in  that  it  contains  a  variable  local  rate 
term  and  also  depends  on  the  digit  reliabilities  computed  by  the  ordering  algo¬ 
rithm.  Finally,  two  other  features  distinguish  this  algorithm  from  both  the 
Fano  and  stack  algorithms.  It  can  declare  a  decoding  failure,  and  its  worst- 
case  computation  is  0(rt~),  where  n  is  the  blocklength. 

Reasons  why  SDR  algorithms  do  not  automatically  outperform  standard 
sequential  decoding  include  the  following.  The  SDR  algorithms  are  con¬ 
strained  to  use  only  low-density  codes,  while  standard  sequential  decoding  can 
use  any  tree  code.  As  discussed  in  Section  1.2,  low-density  code  codeword 
trees  do  not  grow  at  a  constant  rate;  they  tend  to  grow  quickly  near  the  begin¬ 
ning  and  slowly  near  the  end.  This  is  a  drawback  for  reasons  discussed  in 
Section  2.2.  In  addition,  the  features  that  required  the  use  of  a  new  sequen¬ 
tial  decoding  algorithm  make  sequential  decoding  difficult  for  low-density 
codes.  In  addition,  low-density  codes  probably  do  not  have  as  good  distance 
properties  as  arbitrary  tree  codes.  Finally,  the  BSC  is  a  particularly  bad  chan¬ 
nel  for  SDR  algorithms.  Each  digit  at  the  channel  output,  taken  by  itself,  is 
equally  reliable.  As  a  result,  the  ordering  algorithm  does  not  have  much 
information  to  work  with. 

In  conclusion,  the  performance  of  the  SDR  algorithms  with  rate  1/2 
codes  is  encouraging.  The  R0  bound  for  standard  sequential  decoding  is  an 
upper  bound  that  applies  to  all  tree  codes,  while  the  SDR  results  were 


99 


obtained  using  randomly  chosen  codes.  In  addition,  since  the  relative  per¬ 
formance  improved  at  higher  code  rates,  SDR  algorithms  may  outperform 
standard  sequential  decoding  at  rates  greater  than  1/2.  Note  that  if  a  low- 
density  code  with  rate  greater  than  1/2  is  used,  the  code  parameter  j  should 
be  greater  than  2,  as  discussed  in  Section  2.2.  Finally,  the  results  presented 
in  this  work  are  lower  bounds  to  what  can  be  achieved  with  SDR  algorithms. 
Other  ordering  algorithms  may  perform  better. 

4.2  Directions  for  Further  Research 

Most  of  the  results  presented  in  this  work  were  obtained  through  simula¬ 
tions.  More  could  be  said  about  asymptotic  behavior  if  one  obtained  useful 
analytic  bounds  on  the  expected  computation  and  the  probability  of  decoding 
failure.  It  may  be  easier  to  determine  average  values  for  an  ensemble  of  low- 
density  codes,  instead  of  for  a  specific  code. 

Some  more  specific  suggestions  include  the  following.  As  discussed  in 
Section  4.1,  it  appears  promising  that  SDR  algorithms  may  exceed  the  R0 
bound  when  using  code  rates  greater  than  1/2,  and  it  would  be  worthwhile  to 
test  this  hypothesis. 

All  of  the  ordering  algorithms  presented  here,  for  both  the  BEC  and 
BSC,  choose  parity  checks  without  considering  their  effect  on  later  levels  in 
the  codeword  tree.  As  an  alternative,  one  could  consider  c  parity  checks  at  a 
time,  where  c  is  a  constant.  For  example,  the  MNE  algorithm  chooses  parity 
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check  i  to  minimize  M.  (See  Section  2.1.)  Using  this  approach,  the  modi¬ 
fied  MNE  algorithm  would  determine  the  ordered  set  of  c  parity  checks  that 
minimize  Nt+  •  •  •  +  Nf+C_1.  Only  the  first  parity  check  in  the  set  is  chosen, 
so  th~*  each  choice  is  made  with  the  same  amount  of  foresight.  This 
approach  would  require  more  computation  than  the  original  MNE  algorithm, 
but  if  the  performance  improved,  it  would  allow  one  to  trade  off  computation 
in  the  ordering  algorithm  for  computation  in  the  sequential  decoder. 

For  the  BEC,  one  could  use  the  sequential  decoder  described  in  Section 

3.3  instead  of  the  stack  algorithm.  For  the  BSC,  one  could  perform  more 
than  one  iteration  of  Gallager’s  decoding  algorithm  to  generate  digit  reliabili¬ 
ties.  Again,  this  could  be  used  as  a  computation  tradeoff  between  the  order¬ 
ing  algorithm  and  the  sequential  decoder.  Finally,  a  promising  SDR  algorithm 
for  arbitrary  binary  input  memoryless  channels  is  presented  in  Section  4.3. 

4.3  An  SDR  Algorithm  for  Arbitrary  Binary  Input  Memoryless  Channels 

In  this  section,  we  outline  an  SDR  algorithm  that  can  be  used  with  any 
binary  input  memoryless  channel.  This  class  of  channels  includes  the  BEC 
and  BSC,  discussed  in  Chapters  2  and  3,  respectively,  as  well  as  channels 
with  side  information  and  channels  with  real-valued  outputs,  such  as  the  addi¬ 
tive  white  Gaussian  noise  (AWGN)  channel  and  the  Rayleigh  fading  channel. 

Recall  that  SDR  algorithms  consist  of  two  parts  -  an  ordering  algorithm 
and  a  sequential  decoder.  The  ordering  algorithm  used  here,  called  the 


101 


entropy  algorithm,  is  described  below.  The  presentation  loosely  follows  the  C 
programming  language.  First,  we  make  the  following  definitions. 

(n,  j,k)  :  Parameters  of  the  low-density  code  being  used. 

x  =  (,r.)”  :  The  transmitted  codeword. 

y  =  (y.)fl_1  :  The  channel  output. 

m  =  n  j/k  :  The  number  of  parity  checks  used  to  define  the  code. 

{C-,  :  The  parity  check  sets  that  define  the  code,  in  arbitrary 

order.  Each  Ct  C  {1,2 The  parity  equation  corresponding  to  Ct  is 

£^  =  °  (mod  2). 

aec, 

{C- ',  1  <i  <  m }  :  An  ordered  list  of  the  parity  check  sets. 

/  :  A  function  on  the  channel  output  space,  given  by 

f(y„)  =  h(  P(*„  =  0|;O),  (4.i) 

where  h  is  defined  below  and  P  is  the  probability  distribution  generated  by  the 
channel  model  and  the  a  priori  input  distribution.  For  a  channel  with  side 
information,  the  extra  information  would  be  included  together  with  ya  in  the 
formula  for  /. 
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h  :  The  entropy  function,  given  by 


h(x )  =  —x log  x  —  (1— x)Iog  (1— *).  (4.2) 


Input  n,  j,  k,  {Cp  1  <i<nj/k},  y 


Output  {Ci ',  1  <i<n  j/k  } 


$ 

The  Entropy  Algorithm 


m  =  n  j/k ; 

A  =  {1,2 
B  =  {1,2 

for  (i  =  1;  /  <=  m ;  i  =  i+1)  { 

l  =  arg  min  [  £  f(ya)  ]; 

beB  aeC>nA 

c/  =  cr, 

A  =A\Ct; 

B  =B\{iy, 

} 

> 


This  algorithm  has  the  same  structure  as  the  ordering  algorithms  presented  in 
Chapters  2  and  3. 

The  entropy  algorithm  can  be  combined  with  any  sequential  decoding 


algorithm  to  obtain  a  complete  SDR  algorithm.  However,  we  recommend  the 
sequential  decoding  algorithm  described  in  Section  3.3,  which  is  designed 
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specifically  for  low-density  codes.  Pseudocode  for  this  algorithm  is  included 
in  Section  A. 3. 

The  entropy  algorithm  looks  promising  because  it  includes  as  special 
cases  the  MNE  algorithm  (Section  2.1)  for  the  BEC,  and  objective  function 
1.3  (Section  3.1)  for  the  BSC,  both  of  which  were  found  to  work  well.  Also, 
recall  that  one  desirable  objective  of  an  ordering  algorithm  is  to  put  parity 
checks  containing  reliable  new  digits  near  the  front  of  the  codeword  tree. 
The  entropy  algorithm  will  tend  to  satisfy  this  objective,  because  the  entropy 
function  is  a  natural  additive  measure  of  unpredictability.  Finally,  SDR  algo¬ 
rithms  may  work  better  on  channels  with  real-valued  outputs,  such  as  the 
AWGN,  than  on  the  BEC  and  BSC,  because  the  output  can  attain  a  greater 
number  of  reliability  levels.  This  gives  an  ordering  algorithm  more  informa¬ 
tion  to  work  with.  In  a  sense,  the  Vi  statistic,  defined  in  Section  3.1,  is  an 
artificial  way  of  generating  more  reliability  levels  for  the  output  of  the  BSC. 

The  decoding  algorithm  described  here  can  be  used  as  a  soft  decision 
decoder,  because  the  form  of  the  channel  output  is  not  constrained.  Soft 
decision  decoders  usually  perform  better  than  hard  decision  decoders, 
because  less  information  is  thrown  away  by  the  demodulator.  Recall  that  all 
SDR  algorithms,  including  the  one  described  in  this  section,  are  designed  for 
use  with  low-density  codes,  a  class  of  block  codes.  Block  code  decoding 
algorithms  that  handle  soft  decisions  are  uncommon. 
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At  this  point,  we  present  some  potential  improvements  in  the  entropy 

algorithm.  To  begin  with,  it  may  be  advisable  to  quantize  the  function  /. 

This  can  significantly  reduce  the  difficulty  of  evaluating  /.  Next,  define 

E(b )  =  f{ya).  Whether  or  not  we  quantize  /,  there  are  good  reasons 

aecbnA 

for  quantizing  E.  First,  this  can  lead  to  better  orderings,  as  discussed  in  Sec¬ 
tion  3.1.  Second,  we  can  reduce  the  computational  complexity  of  the  entropy 

y 

algorithm  from  0(n“)  to  0(Vn),  where  V  is  the  number  of  distinct  values  £ 
can  assume,  and  j  and  k  are  fixed.  This  reduction  is  accomplished  the  same 
way  as  for  the  BSC  ordering  algorithms  and  is  outlined  in  Section  3.1. 

In  conclusion,  the  decoding  algorithm  presented  here  is  designed  for 
low-density  codes  and  can  be  used  with  any  binary  input  memoryless  channel. 
In  particular,  it  can  be  used  on  channels  with  side  information  and  channels 
with  real-valued  outputs  and  as  a  soft  decision  decoder. 
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APPENDIX 

LIST  OF  ALGORITHMS 

A.l  An  Algorithm  to  Generate  Low-Density  Codes 
A.  1.1  Preliminary  remarks 

The  algorithms  presented  in  the  Appendix  are  described  in  psuedocode, 
using  a  syntax  loosely  based  on  the  C  programming  language.  Some  features 
that  may  be  unfamiliar  to  those  unacquainted  with  C  include  the  following. 
The  operator  “!=”  means  “is  not  equal  to,”  and  “==”  means  “is  equal  to.” 
Arrays  begin  at  index  zero,  unlike  Fortran.  The  presentation  here  differs 
from  C  in  several  ways.  For  example,  the  words  “or”  and  “and”  are  used 
instead  of  “||”  and  “&&.”  Also,  most  argument  lists  for  functions  are  not 
included,  and  some  operations  are  described  in  words  instead  of  computer 
code.  Function  names  are  printed  in  boldface,  and  variables  in  italics. 

The  algorithm  below  attempts  to  generate  the  parity  equations  that  define 

a  low-density  code  with  parameters  (n,  j ,k).  The  quantities  n,  j ,  and  k  must 

be  positive  integers,  with  j  <  k  and  n  a  multiple  of  k.  The  parity  equations 

are  chosen  to  satisfy  the  additional  condition  that  no  two  of  them  involve  the 

same  group  of  two  or  more  digits.  It  may  not  be  possible  to  generate  such  a 

4 

low-density  code  for  given  values  of  n,  /,  and  k.  For  this  reason,  the  algo- 
!  rithm  will  halt  if  calls  >  calLmax.  (A  description  of  all  variables  is  included 

below.) 
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The  algorithm  generates  the  parity  equations  in  blocks  of  size  n/k,  as  dis¬ 
cussed  in  Section  1.3.  Each  digit  appears  in  exactly  one  parity  equation  in 
each  block.  Parity  equations  are  associated  with  rows  in  the  parity  matrix. 
The  algorithm  tries  to  fill  each  row  using  a  random  selection  from  the  avail¬ 
able  digits.  A  digit  is  not  available  if  it  appears  earlier  in  the  current  block, 
or  if  choosing  it  would  cause  two  rows  to  overlap  in  more  than  one  digit.  If 
there  are  no  more  available  digits,  the  algorithm  clears  the  row  and  starts 
over.  It  tries  to  fill  a  row  at  most  r_max  times,  at  which  point  it  starts  over  at 
the  beginning  of  the  current  block.  It  tries  to  fill  a  block  at  most  b_max 
times,  at  which  point  everything  is  cleared  and  the  algorithm  starts  over  from 
scratch.  This  continues  until  either  a  code  is  successfully  generated 
(status  =  0),  or  calls  >  calLmax  (status  —  1).  The  codes  used  in  this  work 
were  generated  using  b_max  =  r_max  —  25. 

The  algorithm  makes  use  of  an  external  function  rand().  It  is  assumed 
that  invoking  rand(smf),  where  seed  is  a  non-zero  floating  point  value,  will 
initialize  a  psuedorandom  number  generator.  Subsequent  calls  of  rand(0.) 
will  return  a  floating  point  value  uniformly  distributed  over  the  interval  [0,1). 
This  corresponds  to  the  function  rand()  provided  with  UNIX  Fortran. 

A.  1.2  Description  of  variables 

b_max  The  maximum  allowed  value  of  block-tries . 

block  The  number  of  the  current  block  being  filled.  Blocks  start  at 

zero. 
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block- failure 

A  flag  that  indicates  block-tries  >  b-max. 

block- filled  A  flag  that  indicates  the  current  block  has  been  successfully 
filled. 

block-tries  The  number  of  times  the  algorithm  has  tried  to  fill  the  current 
block. 

call-max  The  maximum  allowed  value  of  calls. 

calls  The  number  of  times  rand()  is  called,  not  counting  the  initial 

call  of  rand  (seed). 

flag[i ]  A  flag  that  indicates  digit  i  has  been  included  in  a  row  in  the 
current  block. 

flag2[i]  An  array  of  flags  that  indicate  forbidden  digits  for  the  current 
row  being  filled. 

i.  Loop  variables. 

index  The  number  of  digits  included  so  far  in  the  current  row. 

j  A  low-density  code  parameter.  The  code  being  generated  has 

parameters  (n,  j ,k). 

k  Another  low-density  code  parameter. 

n  The  blocklength  of  the  code. 

new-digit  The  new  digit  added  to  the  current  row. 

open  The  number  of  available  digits  that  may  be  added  to  the  current 

row. 

other  [i][s][t  2] 

An  array  used  to  insure  that  no  two  rows  have  more  than  one 
common  digit.  other[i][s][i2]  is  the  index  of  the  i2-nd  digit  in  the 
row  in  block  i  containing  digit  s . 

par[i}[i2]  This  array  stores  the  parity  checks  that  define  the  code  being 
generated,  par [/][/]  is  the  index  of  the  i'-th  digit  involved  in  the 
/-th  parity  check.  Digits  are  numbered  from  0  to  n— 1.  The  first 
parity  check  has  index  one,  not  zero,  in  order  to  be  compatible 
with  the  decoding  algorithm  of  Section  A. 3. 

rand()  An  external  function  that  generates  a  random  number  uniformly 
distributed  over  [0,1).  It  is  described  more  fully  in  the  Prelim¬ 
inary  remarks. 

row  The  number  of  the  current  row  being  filled.  Rows  start  at  zero. 

row-  failure  A  flag  that  indicates  row-tries  >  r-max. 

row-  filled  A  flag  that  indicates  the  current  row  has  been  successfully  filled. 
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row_tries 

s,  s2 

seed 

status 

stop 


The  number  of  times  the  algorithm  has  tried  to  fill  the  current 
row. 

Integer  temporary  variables. 

Initializes  the  random  number  generator. 

The  output  status  of  the  algorithm.  0  =  Code  successfully  gen¬ 
erated,  I  =  Algorithm  stopped  because  calls  >  calLmax. 

A  flag  that  indicates  calls  >  calLmax. 


A.  1.3  Storage  requirements 

All  variables  are  integer  valued,  with  the  exception  of  seed,  which  is  floating 
point  valued.  The  function  rand()  returns  floating  point  values.  The  arrays 
have  the  following  sizes: 

flag[n],  flag2[n],  other[j][n][k],  par[nj/k+ l][k ] 


A.  1.4  The  algorithm 

Input  n,  j,  k,  seed,  b-max ,  r_max,  calLmax 

Output  par,  status,  calls 


Main  program 

{  /*  Begin.  */ 

stop  —  0; 
block  =  0; 

/*  Initialize  random  number  generator.  */ 
rand(seed); 

while  ( block  <  j  and  stop  ==  0)  { 

Construct  a  block; 

if  {block,  filled  ==  1) 
block  =  block +1', 

else 

block  =  0; 


} 
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if  (block  ==  j) 
status  =  0; 

else 

status  =  1; 
}  /*  End.  */ 


Construct  a  block 

{  /*  Begin.  *1 

block-tries  —  0; 
block. _  filled  =  0; 

while  (blocks  filled  ==  0  and  block-tries  <  b-max  and  stop  ==  0)  { 

block-tries  =  block-tries +U 

block- failure  =0; 

row  =  block*n  /k ; 

for  (i  =  0;  i  <  n ;  i  =  i+1) 

//ag[z]  =0; 

while  (row'  <  (Wock+l)*n/fc  and  block- failure  ==  0  and  stop  ==  0)  { 

Construct  a  row; 
if  (row- filled  ==  1) 
row  =  row+1; 

else 

block- failure  =  1; 

> 

if  (row  ==  (block-\-l)*n /k) 
block- filled  —  1; 

} 

return; 


}  /*  End.  */ 
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Construct  a  row 


{  /*  Begin.  */ 

row_tries  =  0; 
row_  filled  =  0; 

while  (row_  filled  ==  0  and  row-tries  <  r_max  and  stop  ==  0)  { 
row_tries  —  row_tries  + 1 ; 


index  —  0; 
row_  failure  =  0; 
for  (i  =  0;  i  <  n;i  —  t'+l) 
flag2[i]  =  0; 

while  (i index  <  k  and  row_  failure  ==  0  and  stop  ==  0)  { 
open  =  0; 

for  (i  =  0,  i  <  n;i  =  i+1) 

if  {flag[i)  ==  0  and  flagl[i]  ==  0) 
open  =  open+1; 

if  {open  >  0)  { 

calls  =  calls + 1; 
if  (ca//s  >=  calLmax ) 
stop  =  1; 

/*  Pick  next  digit.  */ 

/*  The  value  of  s  is  a  random  integer,  uniformly 
distributed  between  1  and  open.  */ 

s  =  integer  part  of  (rand(0)*open+l); 
new_digit  =  —1; 
s2  =  0; 

while  (s 2  <  s)  { 

new_digit  —  new_digit+V, 

if  (flag[new__digit]  ==  0  and  flag2[new_digit ]  ==  0) 
s2  =  s2*hlj 

} 

par[row+l][index]  =  new_digif, 

/*  row+ 1  is  used  because  par  starts  at  one,  not  zero.  */ 


Ill 


I 

I 
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/*  Update  flag2.  */ 

for  (i  =  0;  i  <  block  —  1;  i  —  z  +  1) 

for  (z'2  =  0;  z‘2  <  k ;  z'2  =  z'2+1)  { 
s  =  of/zcr  [z  ][/zew_ziz'gz7  ][/2] ; 
//ag2[j]  =  1; 

> 

flag2[new_digit ]  =  1; 
index  =  index  + 1 ; 


> 

else 

row_ failure  =  1; 

} 

if  (index  ==  k)  { 

/*  This  row  is  complete.  */ 
row_  filled  —  1; 

I*  Update  flag  and  other.  */ 
for  (z  =  0;  i  <  k;i  =  z'+l)  { 

5  =  par  [row + l][z']; 
flag[s]  -  1; 

for  (z'2  =  0;  z'2  <  /c ;  z'2  =  z'2+1) 

oz/zer  [h/oc/c  ][j  ][z  2]  =  par  [ro  w + l][z  2] ; 

> 


> 


> 

return; 


}  /*  End.  V 
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A.2  An  0(n )  Implementation  of  the  MNE  Algorithm 
A. 2.1  Preliminary  remarks 

The  syntax  used  to  present  this  algorithm  is  discussed  in  Section  A.l, 
and  the  MNE  algorithm  is  described  in  Section  2.1.  Briefly,  the  algorithm 
takes  as  input  the  channel  output  vector  and  the  set  of  parity  checks  that 
define  the  low-density  code  being  used.  Initially,  the  parity  checks  are  in  arbi¬ 
trary  order.  The  algorithm’s  purpose  is  to  generate  a  new  ordering,  which  is 
later  used  by  a  separate  sequential  decoding  algorithm  to  generate  a  codeword 
tree. 

Parity  checks  are  referred  to  by  their  position  in  the  original  ordering. 
Each  parity  check  is  associated  with  a  quantity  called  its  e^value ,  defined  to 
be  the  number  of  new  erasures  involved  in  the  parity  check.  The  algorithm 
generates  the  new  ordering  by  choosing  parity  checks  one  at  a  time.  At  each 
step,  it  chooses  the  remaining  parity  check  with  smallest  e_value . 

To  minimize  computation,  the  parity  checks  are  grouped  into  “bins.” 
Each  bin  contains  the  labels  of  all  the  parity  checks  with  a  given  e_value .  To 
start,  the  algorithm  places  each  parity  check  into  the  proper  bin.  Subse¬ 
quently,  the  new  ordering  is  generated  by  choosing  parity  checks  from  the 
first  nonempty  bin  with  smallest  e^value.  However,  when  a  parity  check  is 
chosen,  this  may  change  the  e^value  of  some  remaining  parity  checks.  This 
happens  because  the  definition  of  “new”  erasures  depends  on  which  parity 
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checks  have  already  been  chosen.  As  a  result,  the  algorithm  updates  the  bins 
after  each  choice. 

Each  bin  is  organized  as  a  two-way  linked  list,  with  pointers  stored  in  the 
arrays  forward  and  backward.  Note  forward  and  backward  store  integers, 
not  memory  address  locations;  see  the  Description  of  Variables  below.  The 
array  top  stores  the  location  of  the  most  recent  addition  to  each  bin.  If  a  bin 
is  not  empty,  new  elements  are  placed  at  location  top+ 1.  This  is  done  even  if 
there  are  earlier  empty  locations,  to  save  search  time.  No  bin  is  required  to 
use  more  than  n j/k  locations.  This  happens  because  n j/k  is  the  total 
number  of  parity  checks,  and  no  parity  check  can  enter  a  bin  more  than  once, 
since  a  parity  check’s  e^value  can  never  increase. 


A.2.2  Description  of  variables 
backward[i][i2] 

The  first  nonempty  location  preceding  location  i 2  in  bin  i.  If  z'2 
is  the  first  nonempty  location,  backward[i][i 2]  =  —1. 

bin [/][/ 2]  This  array  holds  the  bins.  The  bin  number  is  i,  and  the  location 
within  the  bin  is  *2. 

e_value[i]  The  number  of  new  erasures  contained  in  parity  check  i. 
exit  A  flag  used  to  control  program  flow. 

flag[i]  An  array  of  flags  that  indicate  which  digits  are  contained  in  the 
parity  checks  chosen  so  far  in  the  new  ordering. 

forward[i][i2] 

The  next  nonempty  location  after  location  i2  in  bin  i.  If  z'2  is 
the  last  nonempty  location,  forward [i][i2]  —  —1. 

i,  i 2,  i 3  Loop  variables. 

j  A  low-density  code  parameter.  The  code  being  used  has  param¬ 

eters  (n,  j,k). 
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k 

label 

leveLmax 

location[i] 

n 

new_top 

next 

old_top 

Par[i][i2] 


A  low-density  code  parameter.  See  / . 

Temporarily  stores  the  contents  of  a  bin  location. 

The  number  of  parity  checks  used  to  define  the  code. 
leveLmax  =  nj/k. 

The  bin  location  of  parity  check  i.  Label  i  is  stored  at 
bin  [e^value  [/ \}[location  [z  ]]. 

The  blocklength  of  the  code. 

The  new  value  of  top  for  the  current  bin. 

The  label  of  the  next  parity  check  in  the  new  ordering. 

The  old  value  of  top  for  the  current  bin. 

This  array  stores  a  list  of  the  parity  checks  that  define  the  code, 
in  the  new  ordering  as  determined  by  the  algorithm.  See  unpar. 


pcj.abel[i][i2 ] 

The  label  of  the  i'2-nd  parity  check  containing  digit  i,  where  z 2 
starts  at  zero. 


rword[i] 


s,  s 2 
sum[i ] 
top[i] 

unpar[i][i2\ 


This  array  stores  the  received  word,  obtained  at  the  output  of 
the  communication  channel.  When  rword[i]  —  2,  this  indicates 
an  erasure  at  digit  i. 

Integer  temporary  variables. 

An  array  of  counters,  used  to  generate  pc_label. 

The  location  of  the  last  element  added  to  bin  /.  top[i ]  =  —  1  if 
bin  i  is  empty. 

This  array  stores  a  list  of  the  parity  checks  that  define  the  code, 
in  the  initial  arbitrary  ordering.  unpar[i][i2]  is  the  index  of  the 
i'2-nd  digit  involved  in  the  z'-th  parity  check.  Parity  checks  are 
numbered  starting  at  one,  not  zero,  to  agree  with  the  presenta¬ 
tion  in  Section  A. 3.  Digits  are  numbered  from  zero  to  n— 1. 


A.2.3  Storage  requirements 

All  variables  are  integer  valued.  The  arrays  have  the  following  sizes,  where 
Im  —  leveLmax +  \  =  n  j/k+1. 

backward[k  +  \][lm],  bin[k+\][lm],  e_value [lm],  flag[n ],  f onvard[k  +  i)[lm], 
loeation[lm ],  par[lm}[k],  pc_label[n][j],  rword[n],  sum[n),  top[k+ 1], 
unpar  [lm][k ] 


A.2.4  The  algorithm 


Input  n,  j,  k,  unpar ,  rword 

Output  par 

Main  Program 

{  /*  Begin  *1 

Initialization; 

for  (z  =  1;  i  <=  leveLmax',  i  —  i+ 1)  { 

/*  Look  for  first  nonempty  bin.  */ 
i2  =  -1; 
exit  —  0; 

while  ( exit  =  0)  { 
i'2=  z'2+1; 
if  (top[i2]  !=  —1) 
exit  ~  1; 

> 

/*  Choose  next  parity  check.  */ 

next  =  Pop(i'2)- 

for  (i2  =  0;  i2  :  k  \  i2—  i'2+l) 

par[i][i2\  =  unpar[next][i2\, 

/*  Update  bins.  */ 

for  (i'2  =  0;  i2  <  k;  i2—  *2+1)  { 

s  =  unpar[next][i2]\ 
if  (flag[s]  ==  0)  { 

flag[s]  =  1; 
if  (rword  [5]  ==  2)  { 

for  (j'3  =  0;  i3  <  j ;  f3=  *3+1)  { 

52  =  pc_label[s][i?>}\ 
if  (52  !=  next )  { 
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Remo  ve(e_value  [52],  location  [52]); 
e_value[s  2]  =  e_value[s  2]— 1; 
location[s  2]  =  Push(52,e_;'aiue[52]); 


> 

}  /*  End  */ 

Initialization 

{  /*  Begin  */ 

5  =/**;; 

leveLmax  —s/k\ 

/*  Computing  leveLmax  in  two  steps  insures  that  multiplication  is 
performed  before  division,  so  truncation  errors  will  not  result  from 
integer  division.  */ 

/*  Initialize  flag .  */ 
for  ( i  =  0;  i  <  n  \  i  =  i+1) 
flag[i]  =  0; 

/*  Generate  pc_label.  */ 
for  (/  =  0;  i  <  n  \  i  =  i+1) 
sum  [i  ]  =  —1; 

for  (i  =  1;  i  <=  leveLmax ;  z  =  z -hi)  { 
for  (i2  =  0;  i2  <  k ;  i2=  i2+l)  { 

5  =  wnpar[i][i2]; 

5wm[5]  =5wm[5]+l; 

pcjlabel  [5  ][sum  [5  ]]  =  i ; 

> 

> 

J*  Initialize  top.  */ 
for  (i  =  0;  i  <=  j;  i  =  i+1) 
top[i]  =  -1; 
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/*  Initialize  bins.  */ 
for  ( i  =  1;  i  <=  leveLmax\  i  =  i+ 1)  { 
s  —0; 

for  (/2  =  0;  i*2  <  k  \  i'2=  t‘2+1) 

if  (rword  [unpar [/  ][/ 2]]  ==  2) 
s  =5+1; 
e_value[i]  =5; 
location[i]  =  Push(5,i); 

> 

}  /*  End  */ 


Push  (s,i) 

/*  Place  parity  check  5  into  bin  i,  and  return  its  location.  */ 
{  /*  Begin  */ 

if  ( top[i ]  ==  -1)  { 

/*  This  bin  is  empty.  *1 

bin[m  ~s’ 

top[i]  =  0; 
forward[i][  0]  =  —1; 
backward[i][  0]  =  —1; 
return  (0); 

> 

else  { 


/*  This  bin  is  not  empty.  */ 

old_top  =top[i ]; 

new_top  =old_top+ 1; 

bin  [i][new_top]  =  5 ; 

top[i]  =  new__top ; 

forward  [i  ][oW_rop  ]  =  new_top ; 

forward  [i  ]  =  —  1; 

backward  [i  ][new_top  ]  =  old_top ; 

return  (new_top ) ; 


> 

>  /*  End  V 
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Pop(/2) 

/*  Remove  the  top  element  from  bin  i‘2,  and  return  its  contents.  */ 
{  /*  Begin  */ 


if  (rop  [/  2]  ==  —1) 
return(— 1); 


else  { 

label  =  bin[i2][top[i2]]; 
new_top  =  backward  [i  2][top  [i  2]] ; 
top[i2]  =  new_top ; 
if  ( new_top  !=  —1) 

=  — 1; 

return  (label)", 


} 

}  /*  End  */ 


Removed,  1 2) 

/*  Remove  the  element  at  location  f2  in  bin  t.  *f 

{  /*  Begin  */ 

if  (top[t]  ==  f2) 

ropjr]  =  backward  [f][f2]; 

if  (/onvarrf[f][f2]  !=  —1) 

Z?tfc&HYzrd[fJ[/0nvard[r][/2]]  =  backward[t][t2\; 

if  ( backward[t][t2 ]  !=  —1) 

forward[t][backward[t][t2]]  =  /orword[t][t2]; 


}  /*  End  */ 
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A.3  A  New  Sequential  Decoding  Algorithm  for  Low-Density  Codes 

A. 3.1  Preliminary  remarks 

Section  3.3  discusses  the  behavior  of  this  algorithm,  and  Section  A.l 
contains  some  notes  on  the  syntax  of  the  description  given  below.  The  algo¬ 
rithm  is  designed  for  the  Binary  Symmetric  Channel,  but  it  can  be  modified  to 
handle  other  binary  input  channels  by  changing  the  formula  for  the  metric,  m. 
With  respect  to  the  simulation  results  of  Section  3.2,  forward  and  lateral 
moves  are  defined  as  follows.  A  forward  or  lateral  move  occurs  whenever  the 
“while”  loop  in  function  Kernel,  presented  below,  is  executed.  When  Kernel 
is  called,  the  first  execution  of  the  while  loop  is  a  forward  move,  and  the 
other  executions  are  lateral  moves.  These  moves  are  functionally  similar  to 
forward  and  lateral  moves  performed  by  the  Fano  sequential  decoding  algo¬ 
rithm  [5].  In  what  follows,  the  word  “level”  refers  to  a  level  in  the  codeword 
tree.  The  terms  n-set  and  o-set  are  defined  in  Section  1.2. 

A.3.2  Description  of  variables 

a[i][i2]  This  array  contains  a  list  of  all  binary  vectors  of  length  k~  1. 
a[i\[i2\  is  digit  *2  of  vector  i. 

active  In  Backtrack  mode,  the  index  of  the  active  parent;  that  is, 

parent[active ]  is  the  parent  being  changed.  Before  any  parent  is 
chosen,  active  =  -1. 

b  The  current  branch  number,  used  in  Kernel. 

b_flag[i][i2]  A  flag  set  to  1  if  branch \parent[i]]  is  set  to  i2  and  the  resulting 
change  is  followed  through  up  to  bt_level. 


best,  best 2  Stores  the  best  change  discovered  in  Backtrack  mode.  This 
change  is  obtained  by  setting  branch\parent[best]]  =  best 2. 

beta  An  intermediate  quantity  used  to  calculate  metric  values. 

branch[l ]  The  branch  number  at  level  l  chosen  by  the  decoder. 

branch_best  The  branch  number  with  metric  value  met-best. 
branch_max[l] 

The  maximum  branch  number  at  level  l.  Branches  are  num¬ 
bered  starting  at  zero,  so  branch_max[l ]  is  one  less  than  the 
number  of  outgoing  branches  leaving  a  node  at  level  /— 1. 

bt_level  The  level  in  the  codeword  tree  from  which  Backtrack  mode  is 
entered. 

cross_ref[i]  The  level  where  digit  i  is  assigned;  in  other  words,  the  level 
where  digit  i  occurs  in  the  n-set. 

d[l]  The  size  of  the  n-set  at  level  /. 

digit[i]  In  Backtrack  mode,  digit[i]  stores  the  value  of  the  digit  in  the  o- 
set  at  btj.evel  that  is  assigned  at  parent[i]. 

exit,  exit!,  exit 3 

Flags  used  to  control  program  flow. 

flag[i]  An  array  of  flags  used  to  generate  new,  old,  and  cross_re f . 

hold[i][i2]  An  array  that  holds  all  possible  values  of  m,  neglecting  the  local 
rate  term.  For  a  given  digit  t,  let  i  =  v[f].  Then  hold[i][ 0]  is 
used  if  x[f]  =  rword[t],  and  hold[i][  1]  is  used  if  x[r]  =£  rword[t ]. 

i,  i 2  Loop  variables. 

j  One  of  the  parameters  of  the  low-density  code  being  used.  The 

code  is  an  ( n,j,k )  low-density  code. 

k  Another  low-density  code  parameter.  See  j. 

I  The  current  level  in  the  codeword  tree.  The  root  node  is  at  level 

zero,  and  the  final  level  is  leveLmax. 

leveLmax  The  final  level  in  the  codeword  tree.  leveLmax  =  n  j/k. 
m[/][/2]  Metric  contribution  for  digit  i  when  x  [i  ]  =  i2. 

met^bc  Stores  the  metric  value  of  the  best  change  discovered  in  Back¬ 
track  mode. 

met_best  The  greatest  branch  metric  encountered  in  the  current  run  of 

Kernel. 

met^comp  When  entering  Backtrack  mode,  stores  met_total [bt_level ] . 
Future  changes  are  compared  to  this  quantity. 

met_total[l ]  Total  path  metric  at  level  l.  met_total[l]  equals  the  sum  of 
metric[i],  from  i  =  1  to  /. 
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met_val[i][i2\ 

The  value  of  met_total[btJ.evel]  that  results  after 
branch\parent[i]}  is  set  to  i2. 

metric [l]  Branch  metric  at  level  l. 

mode  Indicates  the  operating  mode  of  the  decoder.  0  =  Forward 

mode,  1  =  Backtrack  mode,  2  =  Reset  mode. 

n  The  blocklength  of  the  code. 

n_parents  In  Backtrack  mode,  the  total  number  of  parents  of  bt^level. 

new[l][i]  The  i-th  digit  in  the  n-set  at  level  l. 

°ld[l][i]  The  i-th  digit  in  the  o-set  at  level  l. 

old_branch  [i] 

Stores  branch\parent[i]]. 

old_parity[i ]  Stores  parity  {parent [i]]. 

old_set[i]  Stores  set\parent[i]\\parity\parent[i'§[. 

p  The  crossover  probability  of  the  channel. 

p_ flag [z]  A  flag  set  to  1  if  any  branch  setting  is  changed  at  parent]}]. 

par[l][i]  This  array  stores  an  ordered  list  of  the  parity  checks- that  define 

the  code.  par[l}[i]  is  the  index  of  the  i-th  digit  involved  in  the  /- 
th  parity  check.  Digits  are  numbered  from  0  to  n—  1.  The  first 
parity  check  has  index  one,  not  zero,  in  order  to  agree  with  the 
numbering  of  the  codeword  tree  levels. 

parent[i]  In  Backtrack  mode,  this  array  stores  the  locations  of  the  parents 
of  bt_level. 

parity  [l]  The  parity  of  the  o-set  at  level  /.  0  =  even,  1  =  odd. 

r[i]  The  local  code  rate  of  the  codeword  tree,  r [i ]  is  the  local  rate  at 

the  level  which  contains  digit  i  in  its  n-set. 

rword[i]  This  array  stores  the  received  word,  obtained  at  the  output  of 
the  communication  channel,  rword  is  a  binary  vector. 

5,  s2,  s3  Integer  temporary  variables. 

sef[i][i2]  If  sef[i][j'2]  ^  —1,  this  indicates  a  decision  by  the  decoder  to 
assign  branch[i]  =  ser[f][i2]  when  parity  [i]  =  i2. 

status  Records  the  output  status  of  the  decoder.  0  =  Decoding  com¬ 

pleted,  1  =  Decoding  failure  declared,  2  =  Execution  halted 
because  steps  >  step_max . 

steps  The  number  of  forward  and  lateral  moves  performed  by  the 

decoder. 

step_max  The  maximum  number  of  forward  and  lateral  moves  the  decoder 
is  allowed  to  perform. 
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stop  A  flag  set  to  1  when  the  decoder  is  done. 

sx  Floating  point  temporary  variable. 

v[i]  For  rword,  the  number  of  violated  parity  checks  involving  digit  i. 

jt[t]  The  /-th  digit  of  the  codeword  hypothesized  by  the  decoder. 


A.3.3  Storage  requirements 

Note:  n,j,k  are  low-density  code  parameters 

k2  =  2k~l 

Im  —  leveLmax+l  —  nj  /k+1 

int  a[k2][k—  1],  active ,  b,  b-flag[k)[k2],  best,  best2,  branch[lm ], 
branch-best,  branch-max[lm],  bt_level,  cross_re f[n],  d[lm],  digit[k], 
exit,  exit 2,  exit 3,  flag[n],  i,  i2,  j,  k,  level-max ,  mode,  n,  n_parents, 
new[lm][k],  old[lm][k),  old_branch[k],  old_parity[k ],  oldset[k], 

P-flag[k],  par[lm][k],  parent[k],  parity[lm],  rword[n],  s,  s 2,  ^3, 
set[lm][ 2\,  status,  steps,  stepjmax,  stop,  v[n],  x[n}\ 

float  beta,  hold[j+ 1][2],  m[n][2],  met-bc,  met-best,  met-comp, 

met-total[lm],  met_val[k][k2],  metric[lm],  p,  r[n],  sx; 

Note:  m,  met_bc,  met-best,  met_comp ,  met_total,  met_val,  and  metric  must 
be  able  to  store  a  value  of  — oo.  This  can  be  implemented  using  a  correspond¬ 
ing  set  of  one  bit  flags. 


A.3.4  The  algorithm 

Input  n ;  j ;  k ;  par ;  p ;  rword ;  step_max 

Output  x ;  status ;  Steps 


Main  program 


Initialization; 

stop  =  0; 

while  (stop  ==  0)  { 

if  (mode  ==  0)  { 


I*  Forward  mode.  */ 


while  (/  <=  leveLmax  and  mode  ==  0)  { 

Kernel; 

if  (me trie [l]  >=  0) 

/«/+ 1; 

else 

mode  =  1; 

> 

if  (mpde  ==  0)  {  /*  l  >  leveLmax  */ 

status  =  0; 
stop  =  1; 

} 

> 

else  if  ( mode  ==  1)  { 

/*  Backtrack  mode.  */ 

Initialize  backtrack; 

while  (any  untried  parents  left)  { 

Choose  next  parent; 

set[parent[active]][old_parity[active]]  ~  -1; 
p_flag[active]  =  1; 

while  (there  are  any  untried  settings  of  active  parent)  { 

Choose  next  setting; 
if  (parent [active]  <  l ) 

/  =  parent  [active]', 

exit  =  0; 

while  (/  <=  bt_level  and  exit  =-  0)  { 

Kernel; 

if  (l  ==  parent[active ]) 

Digit  test; 

/  =  Z+l; 

} 

if  (exit  ==  0)  { 

s  =  branch  [parent  [active  ]]; 
met_val[active][s ]  =  met_total[bUevel]’, 


124 


b_  flag  [active  }[s]  =  1; 

} 

/  =  parent[active]; 

l*  The  statement  above  has  an  unexpected  but  useful  effect.  If  a  new  parent 
is  chosen,  and  the  new  parent  level  >  the  old  parent  level,  then  the  decoder 
will  first  go  to  the  old  parent  level.  This  allows  the  last  change  at  the  old 
parent  level  to  be  reset  as  the  decoder  moves  forward  to  implement  the  first 
change  at  the  new  parent  level.  */ 

} 


/*  Return  setting  of  active  parent  to  previous  value.  */ 
set[parent[active]][old_parity  [active]]  =  old_set  [active]; 


} 

Decision; 

mode  =  2; 

} 

else  if  (mode  ==  2)  { 

/*  Reset  mode.  */ 
if  (active  !=  -1)  { 

if  (met_bc  >  met_comp)  { 

/*  This  implements  the  best  change  discovered  in  Backtrack  mode.  */ 

set[parent[best]][old_parity[best]]  =  best  2; 
l  =  parent[best]; 


> 

else  { 


I*  In  this  case,  no  change  is  accepted,  and  the  decoder  must  reset 
the  last  change.  */ 


/  =  parent  [active]'. 


} 

while  (/  <=  bt_level)  { 

Kernel; 

l  =1+1; 

} 


> 

mode  =  0; 


} 


Initialization 


5  =  n*j; 
leveLmax  =  s/k; 

/*  Calculating  leveLmax  in  two  steps  insures  that  integer  division 
will  not  cause  truncation  error.  */ 

steps  =  0; 
met_total[0\  =  0; 
mode  =  0; 

/  =  l; 

for  (i  =  1;  i  <=  leveLmax;  i  =  t'+l)  { 

5et[i][0]  =  -1; 

<>er[/][l]  =  -1; 

> 

/*  Calculate  new,  cross_ref ,  old,  d,  branch_max.  */ 
for  (i  =  0:  i  <  n;  i  =  i  +1)  { 
flag[i)  =  0; 

> 

for  (i  =  1;  i  <=  leveLmax;  i  =  i+1)  { 
s  =  0; 

52  =  0; 

for  (/2  =  0;  12  <  k;  12  =  i2+l)  { 

53  »par[i][i2]; 
if  (/teg[53]  ==  0)  { 

5  =5+1; 
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rt6w[i][s]  =53; 
cross_re  /[53]  =  i; 
flag[s  3]  =  1; 

} 

else  { 

52  =52+1; 
old[i][s2)  =  5  3; 

> 

> 

d[i]  =  5; 
if  (5  <=  1) 

branch_max[i ]  =  0; 

else 

branch_max[i ]  =  2i_1—  1; 

} 

/*  Calculate  v.  */ 
for  ( i  =  0;  i  <  n  \  i  =  i+1) 
v[i]  =  0; 

for  (i  =  1;  i  <=  leveLmax ;  i  =  i+1)  { 

if  (rword  does  not  satisfy  parity  check  i)  { 
for  (i2  =  0;  i2  <  fc;  i2  =  t'2+1)  { 

5  =  par[i][i2]; 
v[5]  =  v[5]  +  1; 

> 

} 

} 

/*  Calculate  metric.  */ 

sx  =  (1 — 2p)*~ 1 ; 

beta  =  (1— 5x)/(l+5.x); 

for  ( i  =  0;  i  <=  ;;  i  =  i+1)  { 

sx  =  p*  beta1"2* /{l—p)\ 

/*  With  respect  to  the  notation  of  Section  3.1, 

P(yf = 1^=0  =  i/(i+s*), 

p0f  #  Xt  I  Vt=  i)  =  sx /(1+5X ).  */ 

hold[i][ 0]  =  1  -  log(l+5x)/log(2.); 
hold[i][  1]  =  1  +  log(5x/(l+5*))/log(2.); 

> 


for  (/  =  1;  i  <=  leveLmax ;  i  =  /+1)  { 
if  (d[i]  ==  0)  { 

5  =5+1; 

> 

else  { 

sx  =  d[i]  —  s  —  1; 
sx  =  sx/d[i]\ 

/*  Calculating  sx  in  two  steps  insures  the  division 
is  performed  floating  point,  not  integer.  */ 

5=0; 

for  (i2  =  0;  i2  <  d[i\,  i2  =  i'2+l)  { 

52  =  new[i][i2\, 
r[52]  =  sx; 
if  (rword[s2]  ==  0)  { 

m[52][0]  =  hold[v[s2]}[0]  -  r[s2); 
m[52][l]  =  hold[v[s2])[l]  —  r[52]; 

} 

else  ^ 

m[52][0]  =  hold[v[s2]\[l]  -  r[52]; 
m[52][l]  =  hold[v[s2}}{0]  -  r [52] ; 

} 

> 

} 

> 

return; 


Kernel 

l*  Kernel  determines  branch[l],  and  updates  x,  metric[l], 
and  met_total[l ]  accordingly.  */ 

exit  2  =  0; 
exit  3  =  0; 


/*  Control  returns  to  the  main  program  when  exit 2  is  set  to  1. 
Setting  exit2  =  1  causes  the  while  loop  to  be  executed  exactly 
one  more  time.  *1 
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/*  Calculate  parity  [l].  */ 
s  =  0; 

for  (i  —  0;  /  <  k—d[l}\  i  =  f+1)  { 
if  (old[l][i]  ==  1) 
s  =  1—5 ; 

} 

parity  [l]  —  s\ 

/*  Determine  initial  branch  number.  */ 
if  (set[l]\parity[l})  ==  -1) 
b  =0; 

else  { 

b  =  set[l}\parity[l}\, 
exit  3  =  1; 

} 

while  (exit 2  ==  0)  { 

+1 ; 

if  >  step_max)  { 

status  =  2; 
stop  =  1; 
return; 

> 

if  (d[l)  —  0)  {  _ 

if  (parity  [l  ]  ==  0) 

metric  [l  ]  =  0; 

else 

metric  [l ]  =  —  oo; 
exit  2  —  1; 

} 

else  { 

/*  Generate  branch.  */ 

for  (i  —  0;  i  <  d[l]~  1;  i  ~  /+1)  { 

5  =  new[l][i); 
x[s]  =  a[b][i\; 

Assign  a: [j]  for  5  =  new[l][d[l ]]  so  that 
parity  check  l  is  satisfied; 

/*  Determine  metric  value.  */ 
me  trie  [l]  =  0; 

for  (i  =  0;  i  <  d[l]‘,  i=  i+ 1)  { 


} 


metric  [l]  —  me  trie  [l]  +  m[s][jc[s]]; 


/*  Record  best  branch  so  far.  */ 
if  ( b  ==  0  or  metric [l]  >  met_best )  { 
met-best  =  metric  [/]; 
branch-best  =  b ; 

} 


} 


/*  Decide  whether  to  exit,  and  determine  next  branch  number.  */ 
if  (metric[l]  >=  0  or  exit 3  ==  1) 
exit  2  =  1; 

else  if  ( b  ==  branch_max[l])  { 
exit  3  =  1; 
b  =  branch-best ; 

} 

else 

b  —  b  +1 ; 


}  I*  End  while  loop.  */ 


branch[l ]  =b; 

met_total[l]  =  met_total[l—  1]  +  metric[l]\ 
return; 


Initialize  backtrack 


bt-level  —  l ; 

n-parents  —  k  —  d[bt_level]\ 
met-comp  —  met-total[l]; 
active  =  — 1; 

for  (z  =  0;  i  <  n_parents ;  i=i+ 1)  { 
s  =  old[bt-level][i}', 
parent[i]  =  cross_re  f[s]; 
old-parity  [z  ]  =  parity  [ parent  [i  ]] ; 
old_branch  [z  ]  =  branch  [ parent  [/]]; 
oldset  [i  ]  =  set  [ parent  [z  ]  ][old_parity  [z  ]] ; 
digit  [i]  =x[s]; 

P-flag[i)  =  0; 

for  (z 2  =  0;  z2  <=  branch-max [parent [z]];  z'2  =  z'2+1) 
b-flag[i][i2]  =  0; 
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} 

return; 


Choose  next  parent 

exit  ~  0; 

while  ( active  <  n_parents  and  exit  ==  0)  { 

active  =  active  +1; 
if  (d  [parent  [active]]  >  1) 
exit  =  1; 

/*  If  d[parent[active]]  <=  1,  then  branch^max [parent [active ]]  =0, 
and  so  there  are  no  new  settings  to  try.  Therefore,  a  parent  is 
chosen  only  if  d [parent [active]]  >  1.  */ 


return; 


Choose  next  setting 

/*  Chooses  the  next  setting  of  parent[active ].  */ 

s  =  set  [parent  [active  ]][old_parity  [active  ]] ; 

•  *» 

/*  The  following  statement  causes  the  decoder  to  skip  the  original  setting.  */ 

if  (s  ==  old_branch  [active  ]) 
s  =5+1; 

set  [parent  [active  ]][old~parity  [active  ]]  =  s  +1 ; 

/*  The  condition  in  the  second  while  loop  in  the  main  program  insures  that 
s+1  <  branch_max[parent[active ]].  *! 


return; 
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Digit  test 

/*  If  the  value  of  the  digit  with  index  given  by  old[bt_level][active] 
is  not  changed,  then  the  current  change  at  parent  [active]  will  probably 
not  raise  the  value  of  met_total[bt_level ].  For  this  reason,  the  current 
change  is  aborted  and  the  next  change  is  tried.  */ 

if  (x  [old  [bt_level  ][active  ]]  ==  digit  [active  ]) 
exit  =  1; 
return; 


Decision 

/*  Determines  the  best  change  and  decides  whether  to  declare 
a  decoding  failure.  */ 

met.,  be  =  — oo; 

for  (i  =  0;  i  <  n_parents ;  i  =  i+l)  { 
if  (p~flag[i]  ==  1)  { 

for  (i2  =  0;  i2  <  branch_max\parent[i]\;  i2  =  i'2+l)  { 

if  (b_flag[i][i2]  ==  1  and  met.yal[i][i2]  >  met_.be)  { 
best  —  i ; 
best2  =  j'2; 

met_bc  =  met_val[i][i2}; 

} 

} 

} 

} 

if  (met^comp  ==  — oo  and  met_bc  ==  —  oo)  { 

/*  Decoding  failure.  */ 
status  =  1; 
stop  =  1; 

} 

return; 
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